Sighthound Vehicle Analytics¶

Pipeline Overview¶

VehicleAnalytics pipeline provides vehicle and ALPR detection and classification. The pipeline can utilize a variety of input and output methods, as well as, provides a set of parameters to fine-tune it for needs of a particular application.

The following modules are specific entry-points

VehicleAnalyticsFile.yaml - runs the pipeline on a video or image input file
VehicleAnalyticsRTSP.yaml - runs the pipeline in an RTSP stream
VehicleAnalyticsGstRTSP.yaml - runs the pipeline in an RTSP stream (GStreamer backend, only available on Jetson)
VehicleAnalyticsFolderWatch.yaml - using a configured watched folder as an input
VehicleAnalyticsDirect.yaml - pipeline used by applications embedding SIO as an SDK
VehicleAnalyticsImageLoop.yaml - pipeline feeds a pre-configured image in a loop; useful for performance testing.

Pipeline Parameters¶

Common¶

General¶

detectionMinConfidence - detection confidence threshold. Default: 0.3
detectionOverlapThreshold - parameter to help with filtering detection that have bounding boxes overlapping at least that much. Default: 0.3
detectionModel - specifies the detection model to use. This will affect both performance and accuracy. Values will match the models available in ./models folder, and have a form of generation+[server or embedded]+[standard or fast or best]. So, for example, 6th generation embedded best model will be gen6eb. Default: gen7es
enabledClasses - list of classes enabled in the object detector. Default: ["car", "licenseplate", "bus", "truck", "motorbike"]
imageSaveDir - folder name and prefix where to save analyzed frames corresponding to the generated analytics output. Default: empty (do not generate images)
imageWriterFormat - If set, images will be saved in the specified format. Default: 'jpg'
sourceId - a "callback context" : reported results will include this ID. Useful if gathering outputs from multiple cameras/pipelines at a single point. Default: lpr-1
useTracker - by default, tracker is not utilized, resulting in detected vehicles assuming unique object ID in every frame, unless they are correlated with a license plate. Setting this value to true enables the tracker, at a significant performance cost. We recommend it is never set to true for a stream from a moving source. Default: false
trackerType - choose which algorithm of tracker to use. Options are [cpuBest, cpuFast, cpuFaster, cpuFastest]. Default: cpuFaster
debugSettings - comma-separated list that enables debugging in multiple modules. Supported options: eventAnalytics: Print input and output of event analytics. OSD: Enable OSD streamView events. dumpJson: Save frame_data to the folder specified by debugOutputPath. printJson: Print frameData to stdout. json: Add debug section in frameData. log: Enable debug logging in python modules. filteringDebug: Enable OutputFilter debug. Default: ""
splitMakeModel - if true, the "value" of a car object will be represented as JSON dictionary with values for 'make', 'model', 'generation'. If false, it will be represented as a comma-separated string. Default: false
useComboVehicleModel - if true, use experimental MMY+Color vehicle model. Default: false
comboVehicleModelMode - how to determine MMY outputs. Default: '' (let the pipeline decide)

Update Strategy¶

updateAllowEmptyJSON - if true, output analytics result, even if no objects are contained in it. When false, empty JSON may still be emitted in the case the update is forced due to any of the other update related parameters described here. Default: false
updateMinFrequencyMs - send an output refresh even if no updates had been seen within the specified time from the last output. The update will be sent even if the output is empty. This can be used as a keep-alive mechanism for the client. Default: 0 (Never force updates based on time)
updateOnlyOnChange - if true, only emit new or updated detections (that includes previously detected objects with an improved score). Default: true (except for FolderWatch pipeline, where it is false)
updateOnlyOnEvents - if true, only emit frameData with events ( "sensorEvents" ). Default: false
updateOnMetaclassCountChange - comma-separated list of metaclasses (licensePlates,vehicles) to trigger an update whenever count of that class changes in the output. Default: '' (no updates are triggered by count changes). Special value all means update on count changing for any classes supported by the pipeline.
updateOnStart - send an update at the very start of processing, even if no objects are detected in that frame. Default: true
updateResetStateOnNewInput - relevant for FolderWatch pipeline only; if true, treat each new input as a separate stream. Default: true

License Plate Detection¶

lptPreferAccuracyToSpeed - attempt to cascade and run on alternative crops to get better results. Depending on input, this has yielded up to 3% accuracy improvement at a cost of up to 50% performance loss. Only enable when there is a massive amount of hardware headroom. Default: false
lptModel - specifies the character detection model to use. Same nomenclature applies to this parameter as detectionModel. Default: gen7es.
lptFilter - list of one or more prefefined LP region groups to be considered when classifying license plates. Pre-defined groups are: "us", "eu". Additional groups may be defined by files specified with mmcFilterDefinitions. Default: [ "us" ].
lptFilterDefinitions - path to file specifying user-defined groups of license-plate regions. Default: '' (only system defined groups are considered). Also see Classification Filtering section.
lptMaxCharacters - license plates with a higher number of characters will be ignored. Default: 8
lptMinCharacters - license plates with a lower number of characters will be ignored. Default: 3
lptMinConfidence - confidence score threshold used to filter weightedScore inside the LicensePlateTracker node. Default: 0.5
lptSingleCharThreshold - Confidence score threshold for single character detection. If a character doesn't pass this threshold, the plate will be considered without it. Default: 0.18
lptSkipCarsWithoutLPs - do not report vehicles that aren't associated with a license plate. Default: false
lptStabilizationDelay - Delay output by that many frames, to see if improved result for license plates is available. Default: 0 (do not delay)
lptStabilizationStrategy - What to do, when better result is encountered during stabilization phase: "remove" or "update". Default: "" (system default, 'remove' when updateOnlyOnChange=true, 'update' otherwise)
lptStringMergeWindow - maximum frames to consider when merging license plates objects. Default: 30
lptStringScoring - a way to convert individual character scores to a string score. There are three scoring methods available "min", "max" and "average". The "min" method uses the minimum value in a list of character detection scores as the detection score for a specific license plate string, "max" uses the maximum value, "average" uses the median value. Default: min
lptWeightDetScore - weights for detection and character scores. weightCharScore = 1 - weightDetScore. With default value the pipeline relies more on the character scores. Default: 0.0
rectifyLicensePlates - attempt to de-warp license plates seen under extreme angles. Default: true. It is recommended to keep this enabled, unless camera-to-plate relative position is known to be reasonably straight.

Make, Model, Generation classification¶

mmcFilter - list of one or more prefefined make/mode groups to be considered when classifying license plates. Pre-defined groups are: "us", "eu". Additional groups may be defined by files specified with mmcFilterDefinitions. Default: [ "us" ].
mmcFilterDefinitions - path to file specifying user-defined groups of make/model. Default: '' (only system defined groups are considered). Also see Classification Filtering section.

Classification Filtering¶

Analytics items like license-plate region, vehicle make/model/generation or vehicle color are detected by considering all available choices, and picking the one with the best score. In some cases, those choices need to be heuristically limited based on, for example, locale. System specifies basic groups for both license plate region ('us', eu, 'canada', 'asia'), and make/model ('us', 'eu'). However, sometimes it is useful to limit the choices presented to the pipeline with user-specified criteria. For example, lets say we want to only consider Southeast US license plates as our choices. To accomplish that, create a file (say, /etc/sio/regionGroups.json) with the following content:

{
    "SoutheastUS" : [
        "Florida",
        "Georgia",
        "Alabama"
    ]
}

Start your pipeline adding lptFilter=['SoutheastUS'] lptFilterDefinitions=/etc/sio/regionGroups.json to the launch parameters, and only these states will be considered for the results. Same approach could be applied to make/model/color with mmcFilter/mmcFilterDefinitions parameters. Each file may specify more than one group. To see the full list of available labels, please consult labels.txt of the relevant model's folder.

Box filtering¶

Allows to filter out detection boxes based on bounding box dimensions.

boxFilterConfig - specifies a path to box filter configuration. Default: '' (no filtering). Example configuration:

[
  {
      "name" : "myRoiFilter1",
      # ROI-specific config
      "type" : "roi",
      # The box coordinates are interpreted as `[ x, y, w, h ]`;
      # those coordinates may be absolute pixel coordinates, or relative to the size of the image, with each value being in range of `[0,1]`
      "region" : [ 0.1, 0.1, 0.2, 0.2 ],
      # Classes to apply this filter to
      "classes" : [  "car", "truck" ],
      # boxInRoi (box contained in ROI with minimum of requiredOverlap), roiInBox (opposite)
      "behavior": "boxInRoi",
      # Minimum overlap. Strict > is applied to 0, so specifying that value requires any overlap (Optional, default: 100)
      "requiredOverlap" : 0
      # When set to true, if this filter applies, the object won't pass BoxFilter (Optional, default: false)
      "exclude" : false,
      # Setting this to true will enable additional logging related to this filter
      "debug" : false
  },
  {
      "name" : "mySizeFilter1",
      # Size-specific configuration
      "type" : "size",
      # Value to filter on. Relevant parameters failling outside of constraints will cause the object to be removed from the output
      # Possible values: dimension (either width or height), width, height, area, aspectRatio
      "subtype" : "dimension",
      # Absolute value in pixels, or relative to the frame size, if in (0-1] range
      # 0 means no limit. Both values are optional (defaulting to 0, but if both aren't specified,
      # the filter has no effect)
      "max" : 0.1,
      "min" : 0.1,
      # Classes to apply this filter to
      "classes" : [ "car", "truck" ],
      # Setting this to true will enable additional logging related to this filter
      "debug" : false
  }
]

This can also be useful to turn off unnecessary classifications. For example, setting min of person class to 5000 ensures that no person is ever reported out of the detector, and thus never analyzed, effectively turning off person detection. Similarly, setting all the vehicle constraints to similarly large value can be used to turn off MMC (make model color) analytics.

Extensibility¶

extensionModules - comma-separated list of file paths pointing to Python modules to be used, in order specified, to post-process and potentially modify the JSON output emitted by the pipeline. Default: empty
- Each module specified needs to implement two methods:
  - def configure(configPath) - will be called once, while loading the module. Path to specified configuration will be provided as the parameter.
  - def process(tick, data, frame) - will be called for each frame processed. Parameters are sequential frame ID, json output and RGB frame object. Must return the new value for data.

Disabling specific pipeline components¶

If some output components (make/model/year, color, license plate string, license plate region) are not needed, the following options can be set to avoid analyzing those specific elements:

makeModelDummyValue
colorDummyValue
lpStringDummyValue
lpRegionDummyValue

By default each one of these options is empty, causing the inference to be ran. However, if one is set to a value, the corresponding output will contain that value. Note: if disabling LP analysis by setting both lpStringDummyValue and lpRegionDummyValue, it's advisable to also set rectifyLicensePlates to false

Debugging/Performance¶

debugSettings - enables various debugging facilities. Default: ''. Formatted as a comma separated list of the following values:
- json a debug section is added to the JSON output.
- log enable log output from the pipeline.
- history periodically (and on pipeline termination) log all the reports emitted so far.
debugRawResultsPrefix - when set to a path, will generate raw results useful for post-processing and tuning in the path specified. Default: '' (disabled)
performanceInfo - report performance metrics. Default: false
includeAverageHardware - report average hardware usage information. Default: false
performanceCSVFile - save path for performance metrics CSV, if empty (default) no file will be saved. Default: ''
debugOutputPath - Save path for debug files. Default: '/data/tmp'
debugSaveFormat - save format for output JSONs. Default: 'json'. Options: 'json' or 'csv'

For more detailed information on SIOPerformanceMonitor see, "README_performance.md"

EventAnalytics¶

sensorsConfigFile - path of configuration sensors file, if empty (default) the Event Analytics module will be disabled and input will be forwarded to output. Default: ''

AMQP¶

amqpUseSsl - option to specify if SSL is used. Default: false.
amqpHost - host address for the AMQP output. AMQP output generation will be skipped if this is empty. Default: ''
amqpPort - connection port. Host address and port number make a complete address. Default: 0
amqpMaxRetries - maximum number of connection retries before producing an error message. Default: 5
amqpUser - username that will be used to establish a connection with the host address. Default: ''
amqpPassword - password that will be used to establish a connection with the host address. Default: ''
amqpExchange - used to configure the AMQP Exchange. Default: ''
amqpRoutingKey - used to configure the AMQP Routing key. Default: "sio"
amqpErrorOnFailure - generate an error on AMQP connection/message post failure. Default: false

Pipeline Specific¶

VehicleAnalyticsFile¶

Used for processing video files.

Required Parameters: VIDEO_IN='Video_Path'

Pipeline Specific Options:

outputPrefix - save path and image filename prefix. When set the pipeline wil save images with detection boxes drawn. Default: ''
VIDEO_IN - file to process.

VehicleAnalyticsRTSP¶

Used for processing videos streamed using Real Time Streaming Protocol (RTSP).

Required Parameters: VIDEO_IN='RTSP_URL'

Pipeline Specific Options:

hardwareDecoderType - provides control over the hardware video decoder. Default value results in the use of system default. This can be completely turned off by setting the value to "None". Default: ''
resizeToSize - the size of the images that the pipeline ingests can be controlled using this option. The larger the image the more resource intensive it becomes to process it. A larger source image might also result in more frames dropped during video processing. This value will be used as the maximum height of the frame when it is being resized. Default: 720
maxQueueSize - the maximum number of frames to hold in the frame queue. The default setting for the RTSP pipeline is lossy. Lossy queue will drop frames, rather than stop consuming input when full. When set to 0, asynchronous reading of frames is disabled. Default: 200
fpsLimit - the maximum number of frames that are let through the frame queue. This limits incoming FPS to process to a specific value. This can be used to control resource usage in cases where several streams need to be processed simultaneously. Default: 20
VIDEO_IN - RTSP URI to process.
recordTo - Folder to record the incoming video to. Default: '' (recording disabled)
recordInterval - Interval, in milliseconds, after which a new file will be opened for a recording (may take longer, depending on I-frame distance). Default: 5000
rtspTimeoutMs - RTSP timeout in milliseconds (actual timeout may be a double, if RTSP client attempts to reconnect with a different authentication mechanism). Default: 5000
rtspRetryCount - Number of time RTSP connection will be retried before giving up. Default: 3
rtspUseTCP - Whether to use RTSP over TCP (recommended). Default: true

VehicleAnalyticsGstRTSP¶

Used for processing videos streamed using RTSP. This uses the GStreamer backend which is only available on Jetson.

Required Parameters: VIDEO_IN='RTSP_URL'

Pipeline Specific Options:

fpsLimit - same as the VehicleAnalyticsRTSP pipeline fpsLimit option.
VIDEO_IN - RTSP URI to process.
rtspTimeoutMs - RTSP timeout in milliseconds (actual timeout may be a double, if RTSP client attempts to reconnect with a different authentication mechanism). Default: 5000
rtspRetryCount - Number of time RTSP connection will be retried before giving up. Default: 3

VehicleAnalyticsGstFile¶

Used for processing video files; same as VehicleAnalyticsFile, but using GStreamer rather than ffmpeg-based intake

Required Parameters: VIDEO_IN='Video_Path'

Pipeline Specific Options:

outputPrefix - save path and image filename prefix. When set the pipeline wil save images with detection boxes drawn. Default: ''
VIDEO_IN - file to process.

VehicleAnalyticsFolderWatch¶

Used for processing images using a configured "watched" folder as an input.

Required Parameters: folderPath='Valid_Folder_Path'

Pipeline Specific Options:

folderPath - this is the folder path that will be used to get the input images.
folderPollInterval - this options sets the time interval in milliseconds to wait before checking for new files. Default: 200
folderPollAgeMin - this options sets the time interval in milliseconds for which files are not used, to ensure they're not still being written to storage. Default: 1000
folderPollPrefix - only use files with the prefix set with this option, default is an empty string.
folderPollExtensions - list of image extensions to consider. Default: "["jpg"]"
folderGenerateAnalyticsFile - this option creates a JSON file with the frame data results for each image/video processed in "processed" subfolder. Default: true
folderRemoveSourceFiles - this option when set to true removes source files upon completion, when false, it will move them to "processed" subfolder. Default: false
folderWatchTerminateOnIdle - When no new inputs can be found in the specified folder, the pipeline will terminate. Useful if, and only if, running on a specific contained data set. Default: false
hardwareDecoderType - provides control over the hardware video decoder. This can be completely turned off by setting the value to "None". Default: 'None', due to known problems of nVidia decoder with some types of jpeg.

VehicleAnalyticsDirect¶

This pipeline is used by applications which use SIO as an embedded SDK.

Required Parameters: Pipeline Specific Options:

VehicleAnalyticsImageLoop¶

Used for feeding a pre-configured image in a loop. This is useful for performance testing.

Required Parameters: VIDEO_IN='Image_Path'

Pipeline Specific Options:

VIDEO_IN - path to image to feed as input.