Solid’s test suite is maintained by the Solid test suite panel and sponsored through an Open Collective. You can show your support by donating even just 1 USD there, and we’ll add you or your logo to our list of sponsors.
NB: This does not in any way give you a vote in the contents or the reporting of the test suite, in the Solid spec, or in any aspect of the Solid ecosystem. For that, you should join our W3C-CG.
NLNet Foundation donated 15,000 euros in 2020 as part of the Solid-Nextcloud integration project.
These awesome Solid-related startups collectively sponsor the maintenance of the Solid test suite through our Open Collective, click on their logos to check them out!
And a very big “Thank You” to the following individuals from the Solid community, who are donating through our Open Collective to make the Solid test suite possible, you are all awesome!
|(anonymous backer)||Sjoerd van Groning||Jan Schill||Travis Vachon||Sharon Stratsianis||Matthias Evering|
The reports on solidservers.org are generated using the Jest-based tests, that is, the CRUD tests, the WAC tests, and a few others. Although the Jest-based tests are published by the Solid Test Suite Panel, it is important to note that they have not been reviewed by the Solid spec editors. We are working on migrating to the Gherkin-based specification-tests which can be run using the conformance test harness. Once that migration is complete we will retire the Jest-based tests.
We are also in the process of getting all specification-tests reviewed by spec editors, and until that process is complete we will not publish the output of these test runs directly. Instead, when we see a test failure on specification-tests, we use this as a research observation to do more tests, and if necessary, add a missing test to one of the Jest-based tests so that we can report it on solidservers.org as the output of a Jest-based test.
The more tests you run, the more information you collect. When all tests are green, it confirms what we already thought we knew, and improves our confidence. Even better, especially when test results from different sources contradict each other, this information adds up to help us move forward. This test suite tries to cover all Solid-related protocols and to only test for behaviours that are undisputed in the spec, but it’s evolving and never perfect.
All tests are written from assumptions, and sometimes the same assumption that slipped into your code, also slipped into your tests. In that case, the tests will be green for the wrong reasons. This can be as simple as a typo in a predicate which was maybe copied twice from the same source. Easy to fix, but very important for interoperability!
Sometimes we find a test is incorrect or too strict. Sometimes we don’t know what the correct behaviour is. In this case we mark the test as ‘skip’ and open a spec issue for debate. So at least we can turn an “unknown unknown” into a “known unknown”. When servers disagree, we need to document the difference. If we can describe the differences with reproducable tests, this will help us all have more detailed spec discussions! :)
Is this test suite a single complete and correct source of truth? The answer is no. Solid is still evolving and although there is a lot of consensus around how a Solid pod server should behave, there is no complete single truth. This test suite is an additional layer of defence that will help you compare your implementation of Solid with those of others! That way, we all collectively become more interoperable, and that will ultimately increase the value of Solid for everyone.
See also our chartered deliverables within the Solid community.
Running parts of the test suite against servers can be fiddly, and we’re here to help you. If you have any questions about how to run some tests suite against some server, (whether on your localhost, on the public internet, or in a continuous integration hook like GitHub Actions), please join our Gitter chat for guidance.
The following Solid pod server implementations have been tested in one of three ways; you can tell from the ‘Version’ column:
When a test that is run in one of these three ways fails, we write steps-to-reproduce with curl, and if we are confident the server is violating the spec, we mark it as a fail. If we don’t find any failing tests, we mark the server as ‘✓’. The following table reports our conclusions from that process:
For the ‘version’ column, servers have “(each PR)” if their continuous integration is set up to automatically test against each PR. For closed-source servers we list the public instance against which we run the test suite.
|2.||PHP Solid Server||(each PR)||PHP||✓||7)||✓||✓|
|5.||Community Solid Server||
1) for some servers we have manually tested that they include a working webid-oidc identity provider, but we don’t have the headless-browser tests that confirm this automatically for these servers. The solid-oidc IDP tester page, in contrast, requires human interaction, but with that it can test any publicly hosted IDP.
2) TrinPod will support this in the future
3) Solid 0.9 requires WAC as a “MUST”, yet by default ESS uses ACP instead. ESS can be configured to be compliant with WAC (and thus with Solid 0.9), but this configuration is not enabled on pod.inrupt.com and it is also not supported in production.
4) See #136
5) Due to architectural trade-offs, global locks are not supported in Inrupt ESS
7) PSS supports PATCH with
application/sparql-update but not with the newly required
text/n3, see https://github.com/solid/solid-crud-tests/pull/53/files
8) From our tests it looks like Inrupt Pod Spaces supports PATCH with
application/sparql-update but not with the newly required
text/n3, see #60
When run locally a test-suite-report app can be run :
See latest test-suite-report.md. The report actually covers CRUD and WAC tests of CSS, ESS and NSS.
Version 0.9 of the Solid protocol requires support for WAC, but future versions of the spec will (probably) require servers to support “either WAC or ACP”. We are working on adding test reports for ACP support.
As of 2021, Web Monetization in Solid is an experiment, no real specifications have been written for it yet. These versioned tests are meant to help the discussion as it progresses. The tests themselves are a work in progress, too. More to come as the project progresses. If you’re not yourself working on WebMonetization yourself, don’t spend too much time trying to implement this feature. If youre a Solid app developer and wondering which servers to use when experimenting with WebMonetization in your Solid app, these tests might help you find your way. See https://github.com/solid/monetization-tests.
There is an outdated
runTests.sh script, which is still the best starting point
to run for instance Kjetil’s RDF-based tests, see old-instructions.md.
To run the test suite against the various servers, it’s best to follow their own CI scripts, see the list above.
The scripts are very similar but have small differences in how they start up the system-under-test. One key step is there obtaining a login cookie. Roughly, it works as follows:
www-form-urlencodedPOST to /login/password with
www-form-urlencodedPOST to /login with
Using the cookie, the testers will go through the WebID-OIDC dance (adding the Cookie header in each http request).
This allows the testers to get their DPop tokens signed, which they can then use to construct
We use solid-auth-fetcher for this, specifically its
The webid-provider-tests stop when the tester successfully obtains auth headers.
The solid-crud-tests can run unauthenticated (which is what Community-Solid-Server currently does), or with
The web-access-control-tests have to run authenticated. To pass these tests, the server currently needs to be an identity provider as well as a wac+crud storage. The ‘Alice’ identity on the server should have full R/W/A/C access (accessTo+default) to the entire pod. The tests then instantiate two Solid Logic instances, one for ‘Alice’ on https://server, and one for ‘Bob’ on https://thirdparty. Through those, Alice will edit her ACL documents to give Bob various kinds of access, and then Bob will test various operations.