GPII-1839 Using the Dynamic Device Reporter when running acceptance tests by klown · Pull Request #461 · GPII/universal

klown · 2016-06-24T15:09:33Z

@kaspermarkus One of the work items discussed at the June face-to-face was to use the dynamic device reporter in acceptance testing, and not the mock version. I volunteered to make the changes. I've done two things here:

changed some basic testing config files to specify the device-reporter-in-production-mode as opposed to development mode.
added "deviceReporter" blocks to a number of testSpec files so that the device reporter is run in those tests.

I'm not 100% sure I've found everything that needs changing, but this is ready for review. Let me know if more changes are needed.

Modified "basic" configuration files such that the device reporter is configured/loaded in production mode as opposed to development mode.

Added a "deviceReporter" block to test definitions for the built-in, and orca test specs.

Added a "deviceReporter" block to test definitions for the built-in, jaws, maavis, and nvda test specs.

Fixed lint errors.

gpii-bot · 2016-07-18T17:05:11Z

Could one of the admins verify that these changes are reasonable to test? If so, please reply with "ok to test".

colinbdclark · 2016-07-18T17:05:42Z

ok to test

gpii-bot · 2016-07-19T15:30:50Z

CI job passed.

kaspermarkus

I tried running the NVDA acceptance tests in windows - but on a machine without NVDA installed - and they exploded (since it couldn't find the config file)... I think a good chunk of the purpose of this JIRA is to avoid that the tests explodes if you dont have ALL the 3rd party ATs installed on your device when running acceptance tests. So in my case for example, the acceptance test should pass - but probably print out to the console that NVDA was not installed and that was the reason for passing

kaspermarkus · 2016-09-21T11:02:01Z

        }
    },
-    "mergeConfigs": "%universal/gpii/configs/gpii.config.development.all.local.json"
+    "mergeConfigs": "%universal/gpii/configs/gpii.config.all.development.dr.production.json"


It seems that this file is now identical to "gpii.tests.acceptance.dynamicDeviceReporter.config.json" - so it seems sensible to remove one of them.. perhaps this one and then use the dynamicDeviceReporter instead and update references and docs accordingly

@kaspermarkus I need clarification on what "this one" is. Which two config files are identical? Which one are you suggesting should be deleted? I can find no file with the name you gave above:"gpii.tests.acceptance.dynamicDeviceReporter.config.json".

@kaspermarkus I figured it out. You meant that "gpii.tests.acceptance.localInstall.dynamicDeviceReporter.config.json" is identical to "gpii.tests.acceptance.localInstall.confg.json". I'll delete the latter and replace it with the former, and update docs etc.

@kaspermarkus Done.

kaspermarkus · 2016-09-21T11:05:13Z

        "%flatMatchMaker/configs/gpii.flatMatchMaker.config.development.json",
        "%rawPreferencesServer/configs/gpii.rawPreferencesServer.config.development.json",
-        "%deviceReporter/configs/gpii.deviceReporter.config.development.json"
+        "%deviceReporter/configs/gpii.deviceReporter.config.production.json"


Is this a change that you have made? And if so, what's the reason?

Yes, this is a change I made. The reason being that this JIRA is about using the production version of the device reporter for acceptance testing. I'm guessing from your comment that the proposed change is not specific to testing, or has nothing to do with testing at all. Is that correct?

gpii-bot · 2016-09-22T20:01:59Z

CI job failed. Please visit http://lists.gpii.net/pipermail/ci/ for more details.

klown · 2016-09-29T15:01:08Z

@kaspermarkus Using your fix for GPII-2006 helps, but the tests are still exploding in at least two other places. First, the setting handlers still attempt to set the contents of the INI file to the test values, but an exception is thrown since the file doesn't exist, and the test fails since the actual payload is a set of settings all of which are "undefined". Later, there is an attempt to kill the NVDA process, which isn't running, and that explodes/fails as well.

I'm working on a fix, and looking for similar gotchas.

Don't explode on writing to non-existent settings file.

gpii-bot · 2016-09-29T21:08:58Z

CI job failed. Please visit http://lists.gpii.net/pipermail/ci/ for more details.

amb26 · 2016-11-01T02:26:11Z

 gpii.settingsHandlers.writeFile = function (content, options) {
    if (!options || !options.filename) {
        fluid.fail("writeFile: expected an options block defining filename and encoding");
+    } else if (!fs.existsSync(options.filename)) {


This seems odd - why wouldn't we want to support writing a file which didn't previously exist? I imagine this branch would break the work that @javihernandez would like to do with #462

There are two cases of non-existence. The case here is when the solution is not installed and there is no settings file to write. This is the flip side of using the solution to GPII-2006, where the reading of the file fails for the same reason. @kaspermarkus wrote above:

I tried running the NVDA acceptance tests in windows - but on a machine without NVDA installed - and they exploded (since it couldn't find the config file)...

See also my comment from Sep 24:

... the setting handlers still attempt to set the contents of the INI file to the test values, but an exception is thrown since the file doesn't exist ...

That happens because the settings file is within a directory structure that doesn't exist if the solution is not installed, something like "...\nvda\nvda.ini" There is no parent "nvda" directory. I'll update the unit test for writing to a non-existent file to reflect that.

The other sense of non-existence is where the solution is installed, but needs to be run once to create the settings file. In that case, writing (creating) the settings file should be supported.

We need to handle both cases. The system should be sensitive to the context -- whether the solution is installed -- and react appropriately.

Added some directory structure to better reflect the situation when a settings file does not exist because the solution is not installed.

gpii-bot · 2016-11-01T17:33:38Z

CI job failed. Please visit http://lists.gpii.net/pipermail/ci/ for more details.

- replaced base gpii.tests.acceptance.localInstall.config.json|txt with gpii.tests.acceptance.localInstall.dynamicDeviceReporter.config.json|txt - updated dependant configuration files to reference the new dynamicDeviceReporter base configuration file. - updated all the associated ".txt" documentation files.

gpii-bot · 2016-11-09T15:08:20Z

CI job failed. Please visit http://lists.gpii.net/pipermail/ci/ for more details.

gpii-bot · 2016-11-14T16:09:25Z

CI job failed. Please visit http://lists.gpii.net/pipermail/ci/ for more details.

gtirloni · 2017-02-03T17:49:39Z

ok to test

gpii-bot · 2017-02-03T17:58:38Z

CI job failed. Please visit http://lists.gpii.net/pipermail/ci/ for more details.

javihernandez · 2017-12-13T17:51:10Z

New pull request is #494

klown added 4 commits June 17, 2016 15:52

GPII-1839: Acceptance tests should use the dynamic device reporter.

3b4c3a5

Modified "basic" configuration files such that the device reporter is configured/loaded in production mode as opposed to development mode.

GPII-1839: Linux acceptance tests use the dynamic device reporter.

3290972

Added a "deviceReporter" block to test definitions for the built-in, and orca test specs.

GPII-1839: Windows acceptance tests use the dynamic device reporter.

a61644e

Added a "deviceReporter" block to test definitions for the built-in, jaws, maavis, and nvda test specs.

GPII-1839: Use dynamic device reporter when running acceptance tests.

ee2bc6b

Fixed lint errors.

kaspermarkus self-assigned this Sep 21, 2016

kaspermarkus reviewed Sep 21, 2016

View reviewed changes

Merge master GPII branch into GPII-1839.

69a4128

klown mentioned this pull request Sep 28, 2016

GPII-2006: Now doesn't explode on reading non-existing file #473

Merged

GPII-1839: Merged with latest master.

21e6ddc

GPII-1839: Gracefully handle writing to non-existent file.

378a06a

Don't explode on writing to non-existent settings file.

klown added 3 commits October 3, 2016 10:49

Merge GPII branch 'master' into GPII-1839.

b78dce0

GPII-1839: Merge master GPII branch into GPII-1839.

73226b6

GPII-1839: Merge upstream master GPII branch into GPII-1839.

2faad43

amb26 reviewed Nov 1, 2016

View reviewed changes

GPII-1839: Improved test of handling writing non-existent file.

7c7c6be

Added some directory structure to better reflect the situation when a settings file does not exist because the solution is not installed.

GPII-1839: Merged upstream master GPII branch

55ccf12

javihernandez mentioned this pull request Nov 16, 2016

GPII-2141: Add tests for the device reporter (was: GPII-1310: Add gpii.deviceReporter.alwaysInstalled) #407

Open

javihernandez mentioned this pull request Dec 19, 2016

GPII-1839: Acceptance/Integration tests must use the dynamic device reporter #494

Open

javihernandez closed this Dec 13, 2017

Conversation

klown commented Jun 24, 2016

Uh oh!

gpii-bot commented Jul 18, 2016

Uh oh!

colinbdclark commented Jul 18, 2016

Uh oh!

gpii-bot commented Jul 19, 2016

Uh oh!

kaspermarkus left a comment

Choose a reason for hiding this comment

Uh oh!

kaspermarkus Sep 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

klown Oct 3, 2016

Choose a reason for hiding this comment

Uh oh!

klown Nov 1, 2016

Choose a reason for hiding this comment

Uh oh!

klown Nov 9, 2016

Choose a reason for hiding this comment

Uh oh!

kaspermarkus Sep 21, 2016

Choose a reason for hiding this comment

Uh oh!

klown Oct 3, 2016

Choose a reason for hiding this comment

Uh oh!

gpii-bot commented Sep 22, 2016

Uh oh!

klown commented Sep 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpii-bot commented Sep 29, 2016

Uh oh!

amb26 Nov 1, 2016

Choose a reason for hiding this comment

Uh oh!

klown Nov 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gpii-bot commented Nov 1, 2016

Uh oh!

gpii-bot commented Nov 9, 2016

Uh oh!

gpii-bot commented Nov 14, 2016

Uh oh!

gtirloni commented Feb 3, 2017

Uh oh!

gpii-bot commented Feb 3, 2017

Uh oh!

javihernandez commented Dec 13, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kaspermarkus Sep 21, 2016 •

edited

Loading

klown commented Sep 29, 2016 •

edited

Loading

klown Nov 1, 2016 •

edited

Loading