Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [.................X....X........X.....X............X...X...X.] duration: 23.428504312s FAIL (2025-08-16 15:08:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a76d66aa20905260742a3b4f560d23a09b1c5b91047b45673d8930e17b064963 != before 4160bb81ffbc32d6541fbf6fba1b567fe8ac819affe89325c3a39446eb6aeca8 ======LOGS====== Aug 16 15:07:22.607289776 (189h41m17.038369907s) [upload=197353]: Starting upload Aug 16 15:07:22.607331335 (189h41m17.038411456s) [upload=197353]: Done starting upload Aug 16 15:07:22.607400288 (189h41m17.038480419s) [upload=197353]: Uploading segments... Aug 16 15:07:22.607440354 (189h41m17.038520485s) [upload=197353]: Got next segment. Inline: true Aug 16 15:07:22.607443490 (189h41m17.038523621s) [upload=197353]: Waiting for error group managing segments... Aug 16 15:07:22.607470211 (189h41m17.038550332s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:07:22.607473988 (189h41m17.038554119s) [upload=197353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:07:22.607476713 (189h41m17.038556844s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:07:44.457789165 (189h41m38.888869296s) [upload=197353]: Done uploading segments... Aug 16 15:07:44.457821888 (189h41m38.888902019s) [upload=197353]: Upload finished. err: <nil> Duration:* 23.428531083s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [....................X.X.XXX...X.......X...........X......X..] duration: 44.418597412s FAIL (2025-08-16 15:08:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:05:22.267863991 (189h39m16.698944133s) [upload=197316]: Starting upload Aug 16 15:05:22.267921762 (189h39m16.699001893s) [upload=197316]: Done starting upload Aug 16 15:05:22.268008037 (189h39m16.699088168s) [upload=197316]: Uploading segments... Aug 16 15:05:22.268146161 (189h39m16.699226292s) [upload=197316]: Got next segment. Inline: false Aug 16 15:05:22.268172251 (189h39m16.699252382s) [upload=197316]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:05:22.268176098 (189h39m16.699256229s) [upload=197316]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:05:22.482132489 (189h39m16.91321262s) [upload=197316]: Next segment err: metaclient: Too Many Requests Aug 16 15:05:22.482169078 (189h39m16.913249209s) [upload=197316]: Done uploading segments... Aug 16 15:05:22.482181693 (189h39m16.913261824s) [upload=197316]: Upload finished. err: metaclient: Too Many Requests Duration: 44.418618754s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [....................X.XXX...................................] duration: 756.198086ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:08:01 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 756.223364ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............XXX..XXXXX....................................] duration: 6.909005261s FAIL (2025-08-16 15:07:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.909029586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [..................X....X........X.....X............X...X...X] duration: 24.582345236s FAIL (2025-08-16 15:07:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a76d66aa20905260742a3b4f560d23a09b1c5b91047b45673d8930e17b064963 != before 4160bb81ffbc32d6541fbf6fba1b567fe8ac819affe89325c3a39446eb6aeca8 ======LOGS====== Aug 16 15:07:22.607289776 (189h41m17.038369907s) [upload=197353]: Starting upload Aug 16 15:07:22.607331335 (189h41m17.038411456s) [upload=197353]: Done starting upload Aug 16 15:07:22.607400288 (189h41m17.038480419s) [upload=197353]: Uploading segments... Aug 16 15:07:22.607440354 (189h41m17.038520485s) [upload=197353]: Got next segment. Inline: true Aug 16 15:07:22.607443490 (189h41m17.038523621s) [upload=197353]: Waiting for error group managing segments... Aug 16 15:07:22.607470211 (189h41m17.038550332s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:07:22.607473988 (189h41m17.038554119s) [upload=197353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:07:22.607476713 (189h41m17.038556844s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:07:44.457789165 (189h41m38.888869296s) [upload=197353]: Done uploading segments... Aug 16 15:07:44.457821888 (189h41m38.888902019s) [upload=197353]: Upload finished. err: <nil> Duration:* 24.582366165s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X........XXX..XXXXXX...................................] duration: 6.204730983s FAIL (2025-08-16 15:07:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.204752744s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.XXX..................................] duration: 741.661612ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:07:44 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 741.675228ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [..................XX..XXX...................................] duration: 2.481293844s FAIL (2025-08-16 15:07:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.481316407s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 463.449085ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:07:14 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 463.468522ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X.X.XXX...................................] duration: 12.600999246s FAIL (2025-08-16 15:07:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 12.601027068s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 730.327805ms FAIL (2025-08-16 15:07:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 730.352943ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X........XXX..XXXXXX..................................] duration: 6.261965651s FAIL (2025-08-16 15:06:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.261989826s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [......................X.XXX.................................] duration: 747.525982ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:06:51 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 747.62852ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [...................X....X........X.....X............X...X...] duration: 6.555236159s FAIL (2025-08-16 15:06:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 2e54c5c4ca56e8e5badca302632c1b8a612ccf7ecea3599122fa51eaa00d1fde != before fc11d699dbf92acd805bd1aa63f7d8a42362dbfaccd95d7d3ec1ffd67d94aa77 ======LOGS====== Aug 16 15:05:00.553223749 (189h38m54.984303881s) [upload=197311]: Starting upload Aug 16 15:05:00.553309243 (189h38m54.984389374s) [upload=197311]: Done starting upload Aug 16 15:05:00.553369618 (189h38m54.984449749s) [upload=197311]: Uploading segments... Aug 16 15:05:00.553429193 (189h38m54.984509324s) [upload=197311]: Got next segment. Inline: true Aug 16 15:05:00.553433511 (189h38m54.984513642s) [upload=197311]: Waiting for error group managing segments... Aug 16 15:05:00.553475802 (189h38m54.984555933s) [upload=197311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:05:00.553479789 (189h38m54.98455992s) [upload=197311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:05:00.553482254 (189h38m54.984562375s) [upload=197311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:05:07.544915681 (189h39m1.975995812s) [upload=197311]: Done uploading segments... Aug 16 15:05:07.544934427 (189h39m1.976014558s) [upload=197311]: Upload finished. err: <nil> Duration:* 6.555255868s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 491.996057ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:06:49 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 492.032456ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............XXX..XXXXX...................................] duration: 6.644016391s FAIL (2025-08-16 15:06:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.644045708s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [....................X....X........X.....X............X...X..] duration: 23.602710055s FAIL (2025-08-16 15:06:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 2e54c5c4ca56e8e5badca302632c1b8a612ccf7ecea3599122fa51eaa00d1fde != before fc11d699dbf92acd805bd1aa63f7d8a42362dbfaccd95d7d3ec1ffd67d94aa77 ======LOGS====== Aug 16 15:05:00.553223749 (189h38m54.984303881s) [upload=197311]: Starting upload Aug 16 15:05:00.553309243 (189h38m54.984389374s) [upload=197311]: Done starting upload Aug 16 15:05:00.553369618 (189h38m54.984449749s) [upload=197311]: Uploading segments... Aug 16 15:05:00.553429193 (189h38m54.984509324s) [upload=197311]: Got next segment. Inline: true Aug 16 15:05:00.553433511 (189h38m54.984513642s) [upload=197311]: Waiting for error group managing segments... Aug 16 15:05:00.553475802 (189h38m54.984555933s) [upload=197311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:05:00.553479789 (189h38m54.98455992s) [upload=197311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:05:00.553482254 (189h38m54.984562375s) [upload=197311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:05:07.544915681 (189h39m1.975995812s) [upload=197311]: Done uploading segments... Aug 16 15:05:07.544934427 (189h39m1.976014558s) [upload=197311]: Upload finished. err: <nil> Duration:* 23.60272867s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 472.294267ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:06:31 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 472.31122ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...................XX..XXX..................................] duration: 2.331093315s FAIL (2025-08-16 15:06:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.331115987s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....................X.X.XXX...X.......X...........X......X.] duration: 14.283149387s FAIL (2025-08-16 15:06:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:05:22.267863991 (189h39m16.698944133s) [upload=197316]: Starting upload Aug 16 15:05:22.267921762 (189h39m16.699001893s) [upload=197316]: Done starting upload Aug 16 15:05:22.268008037 (189h39m16.699088168s) [upload=197316]: Uploading segments... Aug 16 15:05:22.268146161 (189h39m16.699226292s) [upload=197316]: Got next segment. Inline: false Aug 16 15:05:22.268172251 (189h39m16.699252382s) [upload=197316]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:05:22.268176098 (189h39m16.699256229s) [upload=197316]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:05:22.482132489 (189h39m16.91321262s) [upload=197316]: Next segment err: metaclient: Too Many Requests Aug 16 15:05:22.482169078 (189h39m16.913249209s) [upload=197316]: Done uploading segments... Aug 16 15:05:22.482181693 (189h39m16.913261824s) [upload=197316]: Upload finished. err: metaclient: Too Many Requests Duration: 14.283175285s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X........XXX..XXXXXX.................................] duration: 6.523104088s FAIL (2025-08-16 15:06:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.523274506s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X.X.XXX..................................] duration: 9.168082586s FAIL (2025-08-16 15:06:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 9.168107245s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 443.685661ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:06:14 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 443.706842ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 782.521543ms FAIL (2025-08-16 15:06:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 782.544989ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.......................X.XXX................................] duration: 744.74927ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:05:52 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 744.771955ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................XXX..XXXXX..................................] duration: 6.952882474s FAIL (2025-08-16 15:05:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.952912581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....................X....X........X.....X............X...X.] duration: 22.210538812s FAIL (2025-08-16 15:05:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 2e54c5c4ca56e8e5badca302632c1b8a612ccf7ecea3599122fa51eaa00d1fde != before fc11d699dbf92acd805bd1aa63f7d8a42362dbfaccd95d7d3ec1ffd67d94aa77 ======LOGS====== Aug 16 15:05:00.553223749 (189h38m54.984303881s) [upload=197311]: Starting upload Aug 16 15:05:00.553309243 (189h38m54.984389374s) [upload=197311]: Done starting upload Aug 16 15:05:00.553369618 (189h38m54.984449749s) [upload=197311]: Uploading segments... Aug 16 15:05:00.553429193 (189h38m54.984509324s) [upload=197311]: Got next segment. Inline: true Aug 16 15:05:00.553433511 (189h38m54.984513642s) [upload=197311]: Waiting for error group managing segments... Aug 16 15:05:00.553475802 (189h38m54.984555933s) [upload=197311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:05:00.553479789 (189h38m54.98455992s) [upload=197311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:05:00.553482254 (189h38m54.984562375s) [upload=197311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:05:07.544915681 (189h39m1.975995812s) [upload=197311]: Done uploading segments... Aug 16 15:05:07.544934427 (189h39m1.976014558s) [upload=197311]: Upload finished. err: <nil> Duration:* 22.21056413s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........X........XXX..XXXXXX................................] duration: 5.998045753s FAIL (2025-08-16 15:05:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.998064028s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 766.067851ms FAIL (2025-08-16 15:05:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 766.096024ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X.X.XXX.................................] duration: 6.97845563s FAIL (2025-08-16 15:05:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 6.97847704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [......................X.X.XXX...X.......X...........X......X] duration: 867.342407ms FAIL (2025-08-16 15:05:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:05:22.267863991 (189h39m16.698944133s) [upload=197316]: Starting upload Aug 16 15:05:22.267921762 (189h39m16.699001893s) [upload=197316]: Done starting upload Aug 16 15:05:22.268008037 (189h39m16.699088168s) [upload=197316]: Uploading segments... Aug 16 15:05:22.268146161 (189h39m16.699226292s) [upload=197316]: Got next segment. Inline: false Aug 16 15:05:22.268172251 (189h39m16.699252382s) [upload=197316]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:05:22.268176098 (189h39m16.699256229s) [upload=197316]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:05:22.482132489 (189h39m16.91321262s) [upload=197316]: Next segment err: metaclient: Too Many Requests Aug 16 15:05:22.482169078 (189h39m16.913249209s) [upload=197316]: Done uploading segments... Aug 16 15:05:22.482181693 (189h39m16.913261824s) [upload=197316]: Upload finished. err: metaclient: Too Many Requests Duration: 867.363728ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................XX..XXX.................................] duration: 2.306226253s FAIL (2025-08-16 15:05:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.306948555s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [........................X.XXX...............................] duration: 771.562986ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:05:15 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 771.587904ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................X....X........X.....X............X...X] duration: 14.304221208s FAIL (2025-08-16 15:05:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 2e54c5c4ca56e8e5badca302632c1b8a612ccf7ecea3599122fa51eaa00d1fde != before fc11d699dbf92acd805bd1aa63f7d8a42362dbfaccd95d7d3ec1ffd67d94aa77 ======LOGS====== Aug 16 15:05:00.553223749 (189h38m54.984303881s) [upload=197311]: Starting upload Aug 16 15:05:00.553309243 (189h38m54.984389374s) [upload=197311]: Done starting upload Aug 16 15:05:00.553369618 (189h38m54.984449749s) [upload=197311]: Uploading segments... Aug 16 15:05:00.553429193 (189h38m54.984509324s) [upload=197311]: Got next segment. Inline: true Aug 16 15:05:00.553433511 (189h38m54.984513642s) [upload=197311]: Waiting for error group managing segments... Aug 16 15:05:00.553475802 (189h38m54.984555933s) [upload=197311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:05:00.553479789 (189h38m54.98455992s) [upload=197311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:05:00.553482254 (189h38m54.984562375s) [upload=197311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:05:07.544915681 (189h39m1.975995812s) [upload=197311]: Done uploading segments... Aug 16 15:05:07.544934427 (189h39m1.976014558s) [upload=197311]: Upload finished. err: <nil> Duration:* 14.304258007s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 454.206069ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:05:12 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 454.23826ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 756.309729ms FAIL (2025-08-16 15:05:09 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 756.338776ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X.X.XXX...X.......X...........X......] duration: 14.860420319s FAIL (2025-08-16 15:05:03 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 16 14:59:25.392574337 (189h33m19.823654468s) [upload=197218]: Starting upload Aug 16 14:59:25.392623722 (189h33m19.823703853s) [upload=197218]: Done starting upload Aug 16 14:59:25.392689177 (189h33m19.823769308s) [upload=197218]: Uploading segments... Aug 16 14:59:25.392790581 (189h33m19.823870702s) [upload=197218]: Got next segment. Inline: false Aug 16 14:59:25.392816010 (189h33m19.823896141s) [upload=197218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:59:25.392819967 (189h33m19.823900098s) [upload=197218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:59:26.229965059 (189h33m20.66104518s) [seg_pos={0 0}, upload=197218]: Begin upload segment... Aug 16 14:59:26.230005056 (189h33m20.661085187s) [seg_pos={0 0}, upload=197218]: Done begin upload segment. Aug 16 14:59:26.230028802 (189h33m20.661108933s) [noise=true , address=104.36.82.51:4080 , piece_id=FEX6WSPOCRSAEDCTGC5XFLPQD64K6YJ6MCT4NBELZQPL7L2ZQFTA, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230048028 (189h33m20.661128149s) [noise=true , address=104.36.82.44:4898 , piece_id=LOCQTZG2FYKD4QUUFCAGAAJZ6FYCTUPGCWE2ASN7Y25UDLVEGOAQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230311252 (189h33m20.661391373s) [noise=true , address=104.36.82.23:4454 , piece_id=5PHOEOGWZBIBE6OINTN6DW2DCQK6AVFXP7AQM5K3AQHHAQHYUPLQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230356909 (189h33m20.66143704s) [noise=true , address=77.90.31.15:4707 , piece_id=CV4FINGKSDB2POM2JML6V7T4D54NL6F36IFQRNMOJLLTZAWJCWBQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230536543 (189h33m20.661616674s) [noise=true , address=66.23.193.175:4556 , piece_id=3CL6ZWCZ4DTYURPHFHEPMJL3BER4ZFAXXPBXNW46GG53VQDBRTVQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230611677 (189h33m20.661691808s) [noise=true , address=104.36.82.60:4360 , piece_id=5YRN7YL3FNS6RJVLMG5DRMBXCFHDNZQUWH22OWHFUZYV2RAH5DNQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230752245 (189h33m20.661832376s) [noise=true , address=45.91.116.8:4072 , piece_id=3I3ZSO5X24532XI5JIUCH3XBNMMJZ3UGWKWFO6NKGQ5AOK6I6YWQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230873116 (189h33m20.661953247s) [noise=true , address=104.36.82.5:4845 , piece_id=BKOUBGO6WVFGJTWUMKOUL6YIQNXP43JITYCLTRRPRRJB2KXDRMLQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230982936 (189h33m20.662063067s) [noise=true , address=104.36.82.35:4600 , piece_id=ZBI4PPN2F4LJH4CAVZQNQUZEOKGBJBXCLPUR5C5ROBJKDTSA5CLQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.231127794 (189h33m20.662207915s) [noise=true , address=79.127.201.215:4355 , piece_id=7O2M6TJ6N43SQKSWNCM2OVOYYQYZEI5DYOCPQHN7MN4NTGFYF5LQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.231365088 (189h33m20.662445219s) [noise=true , address=79.127.201.214:4319 , piece_id=W5FVTCDAJP3MJXNRKLS3SMI4FYT3O7R67HUBJA3VWKNBVLCXSV3Q, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.231584027 (189h33m20.662664158s) [noise=true , address=104.36.82.38:4710 , piece_id=7QLCHTEMCNMM6OTSAWCXJLTNJ4XTBRQN63AICBROVO4HS2XMQYOA, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.231788829 (189h33m20.66286896s) [noise=true , address=77.90.31.21:4892 , piece_id=RXNP7G6TYFO3EF5G3UR2LJFDRZATE7ALULXSA4DSFEE5SGSH4KDQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.231996917 (189h33m20.663077048s) [noise=true , address=77.90.31.11:4568 , piece_id=OSRS7BDJR6RWLS3HLUPJCMKVQECLSYQ2KFBTXCLA72CCTIOESEDQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.232233099 (189h33m20.66331323s) [noise=true , address=104.36.82.37:4688 , piece_id=DFXJWTRZUPFIF5QUJY5ONVXX2K6UJXZMIQ7NVGFRIF6QRVNY3Y7Q, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.232483277 (189h33m20.663563408s) [noise=true , address=104.36.82.36:4649 , piece_id=4V4H4NBVEKZ65G3KXYR3S7WFV7RZNORRLS4BWBDZ7OGWYPMRIRJQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.232689152 (189h33m20.663769283s) [noise=true , address=77.90.31.18:4808 , piece_id=RCXKYFY4DILXKJZASBIXCQKOIFTLZXZWL3QFBPSJNP4QBXKOSFEQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.232889124 (189h33m20.663969255s) [noise=true , address=104.36.82.4:4807 , piece_id=46EACBGRRPNEHVUPJ6VNQ5CX2RVRJ26DNAUFN7QHTEVUBLOWU6NQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:27.336729849 (189h33m21.76780998s) [upload=197218]: Next returned nil segment Aug 16 14:59:27.336745118 (189h33m21.767825249s) [upload=197218]: Waiting for error group managing segments... Aug 16 14:59:27.629928481 (189h33m22.061008612s) [noise=true , address=66.23.193.175:4556 , piece_id=3CL6ZWCZ4DTYURPHFHEPMJL3BER4ZFAXXPBXNW46GG53VQDBRTVQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.185760924 (189h33m22.616841055s) [noise=true , address=79.127.201.214:4319 , piece_id=W5FVTCDAJP3MJXNRKLS3SMI4FYT3O7R67HUBJA3VWKNBVLCXSV3Q, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.288326682 (189h33m22.719406803s) [noise=true , address=104.36.82.44:4898 , piece_id=LOCQTZG2FYKD4QUUFCAGAAJZ6FYCTUPGCWE2ASN7Y25UDLVEGOAQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.306211972 (189h33m22.737292103s) [noise=true , address=79.127.201.215:4355 , piece_id=7O2M6TJ6N43SQKSWNCM2OVOYYQYZEI5DYOCPQHN7MN4NTGFYF5LQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.329176664 (189h33m22.760256785s) [noise=true , address=104.36.82.35:4600 , piece_id=ZBI4PPN2F4LJH4CAVZQNQUZEOKGBJBXCLPUR5C5ROBJKDTSA5CLQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.363114309 (189h33m22.79419444s) [noise=true , address=104.36.82.51:4080 , piece_id=FEX6WSPOCRSAEDCTGC5XFLPQD64K6YJ6MCT4NBELZQPL7L2ZQFTA, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.366686694 (189h33m22.797766825s) [noise=true , address=104.36.82.37:4688 , piece_id=DFXJWTRZUPFIF5QUJY5ONVXX2K6UJXZMIQ7NVGFRIF6QRVNY3Y7Q, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.284993290 (189h33m23.716073411s) [noise=true , address=77.90.31.18:4808 , piece_id=RCXKYFY4DILXKJZASBIXCQKOIFTLZXZWL3QFBPSJNP4QBXKOSFEQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.402227294 (189h33m23.833307425s) [noise=true , address=104.36.82.23:4454 , piece_id=5PHOEOGWZBIBE6OINTN6DW2DCQK6AVFXP7AQM5K3AQHHAQHYUPLQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.415432711 (189h33m23.846512832s) [noise=true , address=77.90.31.15:4707 , piece_id=CV4FINGKSDB2POM2JML6V7T4D54NL6F36IFQRNMOJLLTZAWJCWBQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.873954201 (189h33m24.305034332s) [noise=true , address=104.36.82.36:4649 , piece_id=4V4H4NBVEKZ65G3KXYR3S7WFV7RZNORRLS4BWBDZ7OGWYPMRIRJQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.942769772 (189h33m24.373849903s) [noise=true , address=77.90.31.11:4568 , piece_id=OSRS7BDJR6RWLS3HLUPJCMKVQECLSYQ2KFBTXCLA72CCTIOESEDQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.953987435 (189h33m24.385067566s) [noise=true , address=45.91.116.8:4072 , piece_id=3I3ZSO5X24532XI5JIUCH3XBNMMJZ3UGWKWFO6NKGQ5AOK6I6YWQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:30.080654638 (189h33m24.511734769s) [noise=true , address=77.90.31.21:4892 , piece_id=RXNP7G6TYFO3EF5G3UR2LJFDRZATE7ALULXSA4DSFEE5SGSH4KDQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:30.080681690 (189h33m24.511761821s) [seg_pos={0 0}, upload=197218]: Segment reached optimal threshold of 14 pieces. Aug 16 14:59:30.080946146 (189h33m24.512026267s) [noise=true , address=104.36.82.5:4845 , piece_id=BKOUBGO6WVFGJTWUMKOUL6YIQNXP43JITYCLTRRPRRJB2KXDRMLQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nChJBXktwFs6z7cSK2Wh6N3jkAZwTZ7zhEUt393Nnx4eB6ube): protocol: expected piece hash; context canceled; EOF Aug 16 14:59:30.080958279 (189h33m24.5120384s) [noise=true , address=104.36.82.38:4710 , piece_id=7QLCHTEMCNMM6OTSAWCXJLTNJ4XTBRQN63AICBROVO4HS2XMQYOA, seg_pos={0 0}, upload=197218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JPDQdVWuu5QDXYMAj6MWpaeuKQxdzBdo2t5R7Z25NYXgouscM): protocol: expected piece hash; context canceled; EOF Aug 16 14:59:30.081061306 (189h33m24.512141437s) [noise=true , address=104.36.82.60:4360 , piece_id=5YRN7YL3FNS6RJVLMG5DRMBXCFHDNZQUWH22OWHFUZYV2RAH5DNQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: rpc: dial tcp 104.36.82.60:4360: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:59:30.081070373 (189h33m24.512150505s) [noise=true , address=104.36.82.4:4807 , piece_id=46EACBGRRPNEHVUPJ6VNQ5CX2RVRJ26DNAUFN7QHTEVUBLOWU6NQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MZ9LJLHF4XXvmRG2r7XkanDfdNSKCUgua9EBPusPuGyLzTLo8): protocol: write tcp 188.166.225.170:34916->104.36.82.4:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:59:30.081090081 (189h33m24.512170212s) [seg_pos={0 0}, upload=197218]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 14:59:30.081148193 (189h33m24.512228324s) [upload=197218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 14:59:30.081151789 (189h33m24.51223192s) [upload=197218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:59:30.684221044 (189h33m25.115301165s) [upload=197218]: Done uploading segments... Aug 16 14:59:30.684240732 (189h33m25.115320863s) [upload=197218]: Upload finished. err: <nil> Duration: 14.860447711s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X.X.XXX................................] duration: 6.836477497s FAIL (2025-08-16 15:04:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 6.836503606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................X....X........X.....X............X...] duration: 6.600697874s FAIL (2025-08-16 15:04:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after fbd8f9558d265214e7485d0b5d48ce24f5e668246ab9b2f6bc0e124f609ecc46 != before 67110f220c7aa879f0d8512a7e2c8f78b474e84d1ae5ad935bb0eab7269d06e9 ======LOGS====== Aug 16 15:01:31.045140521 (189h35m25.476220653s) [upload=197251]: Starting upload Aug 16 15:01:31.045190036 (189h35m25.476270157s) [upload=197251]: Done starting upload Aug 16 15:01:31.045249951 (189h35m25.476330082s) [upload=197251]: Uploading segments... Aug 16 15:01:31.045288345 (189h35m25.476368476s) [upload=197251]: Got next segment. Inline: true Aug 16 15:01:31.045293204 (189h35m25.476373335s) [upload=197251]: Waiting for error group managing segments... Aug 16 15:01:31.045328802 (189h35m25.476408933s) [upload=197251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:01:31.045332809 (189h35m25.47641294s) [upload=197251]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:01:31.045335474 (189h35m25.476415605s) [upload=197251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:01:36.571645250 (189h35m31.002725381s) [upload=197251]: Done uploading segments... Aug 16 15:01:36.571664157 (189h35m31.002744288s) [upload=197251]: Upload finished. err: <nil> Duration:* 6.600732681s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........X........XXX..XXXXXX...............................] duration: 5.761925639s FAIL (2025-08-16 15:04:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.76194692s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................XX..XXX................................] duration: 2.394764507s FAIL (2025-08-16 15:04:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.39478732s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................XXX..XXXXX.................................] duration: 6.56189838s FAIL (2025-08-16 15:04:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.56192481s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................X.X.XXX...X.......X...........X.....] duration: 40.197608419s FAIL (2025-08-16 15:04:36 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 16 14:59:25.392574337 (189h33m19.823654468s) [upload=197218]: Starting upload Aug 16 14:59:25.392623722 (189h33m19.823703853s) [upload=197218]: Done starting upload Aug 16 14:59:25.392689177 (189h33m19.823769308s) [upload=197218]: Uploading segments... Aug 16 14:59:25.392790581 (189h33m19.823870702s) [upload=197218]: Got next segment. Inline: false Aug 16 14:59:25.392816010 (189h33m19.823896141s) [upload=197218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:59:25.392819967 (189h33m19.823900098s) [upload=197218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:59:26.229965059 (189h33m20.66104518s) [seg_pos={0 0}, upload=197218]: Begin upload segment... Aug 16 14:59:26.230005056 (189h33m20.661085187s) [seg_pos={0 0}, upload=197218]: Done begin upload segment. Aug 16 14:59:26.230028802 (189h33m20.661108933s) [noise=true , address=104.36.82.51:4080 , piece_id=FEX6WSPOCRSAEDCTGC5XFLPQD64K6YJ6MCT4NBELZQPL7L2ZQFTA, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230048028 (189h33m20.661128149s) [noise=true , address=104.36.82.44:4898 , piece_id=LOCQTZG2FYKD4QUUFCAGAAJZ6FYCTUPGCWE2ASN7Y25UDLVEGOAQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230311252 (189h33m20.661391373s) [noise=true , address=104.36.82.23:4454 , piece_id=5PHOEOGWZBIBE6OINTN6DW2DCQK6AVFXP7AQM5K3AQHHAQHYUPLQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230356909 (189h33m20.66143704s) [noise=true , address=77.90.31.15:4707 , piece_id=CV4FINGKSDB2POM2JML6V7T4D54NL6F36IFQRNMOJLLTZAWJCWBQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230536543 (189h33m20.661616674s) [noise=true , address=66.23.193.175:4556 , piece_id=3CL6ZWCZ4DTYURPHFHEPMJL3BER4ZFAXXPBXNW46GG53VQDBRTVQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230611677 (189h33m20.661691808s) [noise=true , address=104.36.82.60:4360 , piece_id=5YRN7YL3FNS6RJVLMG5DRMBXCFHDNZQUWH22OWHFUZYV2RAH5DNQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230752245 (189h33m20.661832376s) [noise=true , address=45.91.116.8:4072 , piece_id=3I3ZSO5X24532XI5JIUCH3XBNMMJZ3UGWKWFO6NKGQ5AOK6I6YWQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230873116 (189h33m20.661953247s) [noise=true , address=104.36.82.5:4845 , piece_id=BKOUBGO6WVFGJTWUMKOUL6YIQNXP43JITYCLTRRPRRJB2KXDRMLQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.230982936 (189h33m20.662063067s) [noise=true , address=104.36.82.35:4600 , piece_id=ZBI4PPN2F4LJH4CAVZQNQUZEOKGBJBXCLPUR5C5ROBJKDTSA5CLQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.231127794 (189h33m20.662207915s) [noise=true , address=79.127.201.215:4355 , piece_id=7O2M6TJ6N43SQKSWNCM2OVOYYQYZEI5DYOCPQHN7MN4NTGFYF5LQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.231365088 (189h33m20.662445219s) [noise=true , address=79.127.201.214:4319 , piece_id=W5FVTCDAJP3MJXNRKLS3SMI4FYT3O7R67HUBJA3VWKNBVLCXSV3Q, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.231584027 (189h33m20.662664158s) [noise=true , address=104.36.82.38:4710 , piece_id=7QLCHTEMCNMM6OTSAWCXJLTNJ4XTBRQN63AICBROVO4HS2XMQYOA, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.231788829 (189h33m20.66286896s) [noise=true , address=77.90.31.21:4892 , piece_id=RXNP7G6TYFO3EF5G3UR2LJFDRZATE7ALULXSA4DSFEE5SGSH4KDQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.231996917 (189h33m20.663077048s) [noise=true , address=77.90.31.11:4568 , piece_id=OSRS7BDJR6RWLS3HLUPJCMKVQECLSYQ2KFBTXCLA72CCTIOESEDQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.232233099 (189h33m20.66331323s) [noise=true , address=104.36.82.37:4688 , piece_id=DFXJWTRZUPFIF5QUJY5ONVXX2K6UJXZMIQ7NVGFRIF6QRVNY3Y7Q, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.232483277 (189h33m20.663563408s) [noise=true , address=104.36.82.36:4649 , piece_id=4V4H4NBVEKZ65G3KXYR3S7WFV7RZNORRLS4BWBDZ7OGWYPMRIRJQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.232689152 (189h33m20.663769283s) [noise=true , address=77.90.31.18:4808 , piece_id=RCXKYFY4DILXKJZASBIXCQKOIFTLZXZWL3QFBPSJNP4QBXKOSFEQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:26.232889124 (189h33m20.663969255s) [noise=true , address=104.36.82.4:4807 , piece_id=46EACBGRRPNEHVUPJ6VNQ5CX2RVRJ26DNAUFN7QHTEVUBLOWU6NQ, seg_pos={0 0}, upload=197218]: Uploading piece... Aug 16 14:59:27.336729849 (189h33m21.76780998s) [upload=197218]: Next returned nil segment Aug 16 14:59:27.336745118 (189h33m21.767825249s) [upload=197218]: Waiting for error group managing segments... Aug 16 14:59:27.629928481 (189h33m22.061008612s) [noise=true , address=66.23.193.175:4556 , piece_id=3CL6ZWCZ4DTYURPHFHEPMJL3BER4ZFAXXPBXNW46GG53VQDBRTVQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.185760924 (189h33m22.616841055s) [noise=true , address=79.127.201.214:4319 , piece_id=W5FVTCDAJP3MJXNRKLS3SMI4FYT3O7R67HUBJA3VWKNBVLCXSV3Q, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.288326682 (189h33m22.719406803s) [noise=true , address=104.36.82.44:4898 , piece_id=LOCQTZG2FYKD4QUUFCAGAAJZ6FYCTUPGCWE2ASN7Y25UDLVEGOAQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.306211972 (189h33m22.737292103s) [noise=true , address=79.127.201.215:4355 , piece_id=7O2M6TJ6N43SQKSWNCM2OVOYYQYZEI5DYOCPQHN7MN4NTGFYF5LQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.329176664 (189h33m22.760256785s) [noise=true , address=104.36.82.35:4600 , piece_id=ZBI4PPN2F4LJH4CAVZQNQUZEOKGBJBXCLPUR5C5ROBJKDTSA5CLQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.363114309 (189h33m22.79419444s) [noise=true , address=104.36.82.51:4080 , piece_id=FEX6WSPOCRSAEDCTGC5XFLPQD64K6YJ6MCT4NBELZQPL7L2ZQFTA, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:28.366686694 (189h33m22.797766825s) [noise=true , address=104.36.82.37:4688 , piece_id=DFXJWTRZUPFIF5QUJY5ONVXX2K6UJXZMIQ7NVGFRIF6QRVNY3Y7Q, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.284993290 (189h33m23.716073411s) [noise=true , address=77.90.31.18:4808 , piece_id=RCXKYFY4DILXKJZASBIXCQKOIFTLZXZWL3QFBPSJNP4QBXKOSFEQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.402227294 (189h33m23.833307425s) [noise=true , address=104.36.82.23:4454 , piece_id=5PHOEOGWZBIBE6OINTN6DW2DCQK6AVFXP7AQM5K3AQHHAQHYUPLQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.415432711 (189h33m23.846512832s) [noise=true , address=77.90.31.15:4707 , piece_id=CV4FINGKSDB2POM2JML6V7T4D54NL6F36IFQRNMOJLLTZAWJCWBQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.873954201 (189h33m24.305034332s) [noise=true , address=104.36.82.36:4649 , piece_id=4V4H4NBVEKZ65G3KXYR3S7WFV7RZNORRLS4BWBDZ7OGWYPMRIRJQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.942769772 (189h33m24.373849903s) [noise=true , address=77.90.31.11:4568 , piece_id=OSRS7BDJR6RWLS3HLUPJCMKVQECLSYQ2KFBTXCLA72CCTIOESEDQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:29.953987435 (189h33m24.385067566s) [noise=true , address=45.91.116.8:4072 , piece_id=3I3ZSO5X24532XI5JIUCH3XBNMMJZ3UGWKWFO6NKGQ5AOK6I6YWQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:30.080654638 (189h33m24.511734769s) [noise=true , address=77.90.31.21:4892 , piece_id=RXNP7G6TYFO3EF5G3UR2LJFDRZATE7ALULXSA4DSFEE5SGSH4KDQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: <nil> Aug 16 14:59:30.080681690 (189h33m24.511761821s) [seg_pos={0 0}, upload=197218]: Segment reached optimal threshold of 14 pieces. Aug 16 14:59:30.080946146 (189h33m24.512026267s) [noise=true , address=104.36.82.5:4845 , piece_id=BKOUBGO6WVFGJTWUMKOUL6YIQNXP43JITYCLTRRPRRJB2KXDRMLQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nChJBXktwFs6z7cSK2Wh6N3jkAZwTZ7zhEUt393Nnx4eB6ube): protocol: expected piece hash; context canceled; EOF Aug 16 14:59:30.080958279 (189h33m24.5120384s) [noise=true , address=104.36.82.38:4710 , piece_id=7QLCHTEMCNMM6OTSAWCXJLTNJ4XTBRQN63AICBROVO4HS2XMQYOA, seg_pos={0 0}, upload=197218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JPDQdVWuu5QDXYMAj6MWpaeuKQxdzBdo2t5R7Z25NYXgouscM): protocol: expected piece hash; context canceled; EOF Aug 16 14:59:30.081061306 (189h33m24.512141437s) [noise=true , address=104.36.82.60:4360 , piece_id=5YRN7YL3FNS6RJVLMG5DRMBXCFHDNZQUWH22OWHFUZYV2RAH5DNQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: rpc: dial tcp 104.36.82.60:4360: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:59:30.081070373 (189h33m24.512150505s) [noise=true , address=104.36.82.4:4807 , piece_id=46EACBGRRPNEHVUPJ6VNQ5CX2RVRJ26DNAUFN7QHTEVUBLOWU6NQ, seg_pos={0 0}, upload=197218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MZ9LJLHF4XXvmRG2r7XkanDfdNSKCUgua9EBPusPuGyLzTLo8): protocol: write tcp 188.166.225.170:34916->104.36.82.4:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:59:30.081090081 (189h33m24.512170212s) [seg_pos={0 0}, upload=197218]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 14:59:30.081148193 (189h33m24.512228324s) [upload=197218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 14:59:30.081151789 (189h33m24.51223192s) [upload=197218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:59:30.684221044 (189h33m25.115301165s) [upload=197218]: Done uploading segments... Aug 16 14:59:30.684240732 (189h33m25.115320863s) [upload=197218]: Upload finished. err: <nil> Duration: 40.197675637s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 465.803089ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:04:17 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 465.826995ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 733.437665ms FAIL (2025-08-16 15:04:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 733.465097ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................XXX..XXXXX................................] duration: 6.909104989s FAIL (2025-08-16 15:04:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.909134265s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.........................X.XXX..............................] duration: 762.140066ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:04:03 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 762.16856ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [........................X....X........X.....X............X..] duration: 14.120654119s FAIL (2025-08-16 15:03:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after fbd8f9558d265214e7485d0b5d48ce24f5e668246ab9b2f6bc0e124f609ecc46 != before 67110f220c7aa879f0d8512a7e2c8f78b474e84d1ae5ad935bb0eab7269d06e9 ======LOGS====== Aug 16 15:01:31.045140521 (189h35m25.476220653s) [upload=197251]: Starting upload Aug 16 15:01:31.045190036 (189h35m25.476270157s) [upload=197251]: Done starting upload Aug 16 15:01:31.045249951 (189h35m25.476330082s) [upload=197251]: Uploading segments... Aug 16 15:01:31.045288345 (189h35m25.476368476s) [upload=197251]: Got next segment. Inline: true Aug 16 15:01:31.045293204 (189h35m25.476373335s) [upload=197251]: Waiting for error group managing segments... Aug 16 15:01:31.045328802 (189h35m25.476408933s) [upload=197251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:01:31.045332809 (189h35m25.47641294s) [upload=197251]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:01:31.045335474 (189h35m25.476415605s) [upload=197251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:01:36.571645250 (189h35m31.002725381s) [upload=197251]: Done uploading segments... Aug 16 15:01:36.571664157 (189h35m31.002744288s) [upload=197251]: Upload finished. err: <nil> Duration:* 14.120674598s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........X........XXX..XXXXXX..............................] duration: 5.775174094s FAIL (2025-08-16 15:03:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.775196818s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................XX..XXX...............................] duration: 2.259773629s FAIL (2025-08-16 15:03:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.259793328s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)