Here is a lightning post describing how to add a nice 404 error to your S3 hosted Pelican site.
Add the page to pelican.
This is sufficiently described in the docs.
Identify 404 resource in s3.
This is also very easy, with the setup as suggested in the section above, point to the 404 page from the AWS bucket static website hosting menu. It should look like the image below, provided you used the setup described in the pelican docs.