Statistics not working after upgrade

[Richard]

We recently upgraded our ad server from OpenX 2.8 to Revive 3.2.5 by following the instructions at https://www.revive-adserver.com/support/upgrading/

However, since that time we've not had any statistics recorded since. I've gone through https://documentation.revive-adserver.com/display/DOCS/No+Statistics and verified that everything there is correct except that the table described in Step 2 doesn't exist. I did find the ox_data_bkt_m table which I assume is the same one and was just never renamed during the upgrade. However, this table does not contain any data.

I verified that everything else under Step 2 is correct with the one exception that the openXMarket plugin is missing. I understand that this should be removed now as it is no longer needed, but the Uninstall button just fails with Failed to find package definition file <ad-server root>/plugins/etc/openXMarket.xml.

I've double-checked the invocation tags and they are correct according to the interface. I've also double-checked the maintenance settings to ensure the maintenance script is running.

I looked at the logs and that seems to be more or less correct (shown below). Looking through the logs, apart from an error regarding this plugin that I can't get rid of, there is some sort of issue with the ox_data_bkt_a and ox_data_bkt_a_var tables but I'm not sure how to fix them.

I also note that there is a line in the log Apr 24 00:00:51 +0000 OX-maintenance-58fd40039e37a [     info]  Inserted 15604 rows in the table tmp_ad_zone_impression which suggests that stats are being collected but I'm unable to find this table.

Can anyone suggest what I should do now? Or is there something I missed somewhere along the way?

 

Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Running Maintenance Engine
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Running Maintenance Statistics Engine
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [    error]  Failed to find package definition file /home/dlook/software/ads/revive-adserver-3.2.5/plugins/etc/openXMarket.xml
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OX_Maintenance_Statistics_Task_SetUpdateRequirements
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  - Maintenance statistics will be run
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  - The intermediate table statistics will be updated
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  - The final table statistics will be updated
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OX_Maintenance_Statistics_Task_SetUpdateRequirements
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OX_Maintenance_Statistics_Task_MigrateBucketData
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [  warning]  Invalid argument supplied for foreach()
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]      - The ox_data_bkt_a table is a raw data table. Data logged in real-time, not operation intervals.
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]      - Accordingly, pruning of the ox_data_bkt_a table will be performed based on data that has a logged date between
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]        2017-04-23 23:00:00 UTC and 2017-04-23 23:59:59 UTC
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]      - The ox_data_bkt_a_var table is a raw data table. Data logged in real-time, not operation intervals.
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]      - Accordingly, pruning of the ox_data_bkt_a_var table will be performed based on data that has a logged date between
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]        2017-04-23 23:00:00 UTC and 2017-04-23 23:59:59 UTC
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OX_Maintenance_Statistics_Task_MigrateBucketData
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OX_Maintenance_Statistics_Task_DeDuplicateConversions
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OX_Maintenance_Statistics_Task_DeDuplicateConversions
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OX_Maintenance_Statistics_Task_ManageConversions
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OX_Maintenance_Statistics_Task_ManageConversions
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OX_Maintenance_Statistics_Task_SummariseFinal
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OX_Maintenance_Statistics_Task_SummariseFinal
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OX_Maintenance_Statistics_Task_LogCompletion
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OX_Maintenance_Statistics_Task_LogCompletion
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OX_Maintenance_Statistics_Task_ManageCampaigns
Apr 24 00:00:03 +0000 OX-maintenance-58fd40039e37a [     info]  - Managing (activating/deactivating) campaigns
Apr 24 00:00:47 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OX_Maintenance_Statistics_Task_ManageCampaigns
Apr 24 00:00:47 +0000 OX-maintenance-58fd40039e37a [     info]  Maintenance Statistics Engine Completed (Started at 2017-04-24 00:00:03 UTC)
Apr 24 00:00:47 +0000 OX-58fd40039e37a [     info]  Running Midnight Maintenance Tasks
Apr 24 00:00:48 +0000 OX-58fd40039e37a [     info]  Begin pruning old records from data_summary_ad_zone_assoc
Apr 24 00:00:48 +0000 OX-58fd40039e37a [     info]  Finished pruning old records from data_summary_ad_zone_assoc: 0 records deleted
Apr 24 00:00:48 +0000 OX-58fd40039e37a [     info]  Begin pruning records for expired inactive campaigns from data_summary_ad_zone_assoc
Apr 24 00:00:48 +0000 OX-58fd40039e37a [     info]  Finished pruning expired inactive campaigns from data_summary_ad_zone_assoc: 0 records deleted
Apr 24 00:00:48 +0000 OX-58fd40039e37a [     info]  Begin pruning records for completed inactive campaigns from data_summary_ad_zone_assoc
Apr 24 00:00:48 +0000 OX-58fd40039e37a [     info]  Table ox_data_summary_ad_zone_assoc overhead (number of allocated but unused bytes) = unkown
Apr 24 00:00:48 +0000 OX-58fd40039e37a [     info]  Midnight Maintenance Tasks Completed
Apr 24 00:00:48 +0000 OX-maintenance-58fd40039e37a [     info]  Running Maintenance Priority Engine
Apr 24 00:00:48 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OA_Maintenance_Priority_AdServer_Task_GetRequiredAdImpressionsLifetime
Apr 24 00:00:48 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OA_Maintenance_Priority_AdServer_Task_GetRequiredAdImpressionsLifetime
Apr 24 00:00:48 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OA_Maintenance_Priority_AdServer_Task_GetRequiredAdImpressionsDaily
Apr 24 00:00:48 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OA_Maintenance_Priority_AdServer_Task_GetRequiredAdImpressionsDaily
Apr 24 00:00:48 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OA_Maintenance_Priority_AdServer_Task_AllocateZoneImpressions
Apr 24 00:00:48 +0000 OX-maintenance-58fd40039e37a [     info]    - Selecting Zones forecasts for all managers for OI from 2017-04-24 00:00:00 to 2017-04-24 00:59:59
Apr 24 00:00:51 +0000 OX-maintenance-58fd40039e37a [     info]  Inserted 15604 rows in the table tmp_ad_zone_impression
Apr 24 00:00:51 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OA_Maintenance_Priority_AdServer_Task_AllocateZoneImpressions
Apr 24 00:00:51 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OA_Maintenance_Priority_AdServer_Task_PriorityCompensation
Apr 24 00:00:51 +0000 OX-maintenance-58fd40039e37a [     info]    - Selecting Zones forecasts for all managers for OI from 2017-04-24 00:00:00 to 2017-04-24 00:59:59
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OA_Maintenance_Priority_AdServer_Task_PriorityCompensation
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OA_Maintenance_Priority_AdServer_Task_ECPMforContract
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]  - Selecting All Managers
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]    - Selecting active campaigns with priority = '9' for manager 1
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]    - Selecting active campaigns with priority = '8' for manager 1
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]    - Selecting active campaigns with priority = '7' for manager 1
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]    - Selecting active campaigns with priority = '6' for manager 1
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OA_Maintenance_Priority_AdServer_Task_ECPMforContract
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]  Task begin: OA_Maintenance_Priority_AdServer_Task_ECPMforRemnant
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]  - Selecting Managers with ECPM priority campaigns running
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]  Task complete: OA_Maintenance_Priority_AdServer_Task_ECPMforRemnant
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]  Maintenance Priority Engine Completed (Started at 2017-04-24 00:00:03 UTC)
Apr 24 00:00:52 +0000 OX-maintenance-58fd40039e37a [     info]  Maintenance Engine Completed (Started at 2017-04-24 00:00:03 UTC, taking 00:00:49)
Apr 24 00:00:52 +0000 OX-58fd40039e37a [     info]  Delaying Revive Adserver sync process by 29 seconds.
Apr 24 00:01:25 +0000 OX-58fd40039e37a [     info]  Sync: updates found!
Apr 24 00:01:25 +0000 OX-58fd40039e37a [     info]  Finished Revive Adserver Sync process.
 

[andrewatfornax]

On 4/24/2017 at 1:10 PM, Richard said:

However, since that time we've not had any statistics recorded since. I've gone through https://documentation.revive-adserver.com/display/DOCS/No+Statistics and verified that everything there is correct except that the table described in Step 2 doesn't exist. I did find the ox_data_bkt_m table which I assume is the same one and was just never renamed during the upgrade. However, this table does not contain any data.

 

Hi @Richard,

Okay, that's the problem. We don't re-name the database tables on upgrade, as the prefix (rv_ by default now, but ox_ in your case) doesn't really matter. So, for you, ox_data_bkt_m is where stats should be recorded on delivery.

The fact that they are not there indicates that during delivery, Revive Adserver can't log the statistics.

Take a look at steps 2.1, 2.2 and 2.3 again, now that we know this is the problem point?

[Richard]

Hi @andrewatfornax,

Thanks for the reply. 

You've hit the nail on the head. Based on the steps, I agree that would seem to be the point at where the problem is occurring.

However,

1. The Banner Delivery Logging Plugin is both installed and enabled
2. I've double-checked the Banner Logging Settings and both impression and click logging is enabled. Nor have any restrictions at all been placed on what is logged.
3. I've also confirmed that the database user we've setup for the AdServer can insert data (both generally and specifically to the ox_data_bkt_m table.

So, I'm left scratching my head as to what the problem could be. I've confirmed that the settings are correct according to the various guides and documents .

It suggests that there is a bug somewhere but I can't figure out what it could be or where it is.

Is there any sort of more verbose logging that I can enable in order to see what is going on at each stage of the process? Or a document that describes in detail what should happen where and how I might be able to verify it to see if we can trace the source of the issue?

Thanks,Richard.

Hi @andrewatfornax,

Thanks for the reply. 

You've hit the nail on the head. Based on the steps, I agree that would seem to be the point at where the problem is occurring.

However,

1. The Banner Delivery Logging Plugin is both installed and enabled
2. I've double-checked the Banner Logging Settings and both impression and click logging is enabled. Nor have any restrictions at all been placed on what is logged.
3. I've also confirmed that the database user we've setup for the AdServer can insert data (both generally and specifically to the ox_data_bkt_m table.

So, I'm left scratching my head as to what the problem could be. I've confirmed that the settings are correct according to the various guides and documents .

It suggests that there is a bug somewhere but I can't figure out what it could be or where it is.

Is there any sort of more verbose logging that I can enable in order to see what is going on at each stage of the process? Or a document that describes in detail what should happen where and how I might be able to verify it to see if we can trace the source of the issue?

Thanks,Richard.

Edited May 1, 2017 by Richard

[andrewatfornax]

Hi @Richard,

Yeah, in the configuration file, under the [deliveryLog] section is an option to enable delivery logging. Watch the load that this will introduce, though.

See if that helps - if not, then it might be a case of needing to get xdebug installed so that you can trace through the execution of the code.