Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X........X........X.X...XXX.........X......X....X..X......] duration: 36.478329797s FAIL (2025-07-09 21:16:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.478343553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.X..X..X..X..X..X....X..X..........X...X................X...] duration: 45.095442657s FAIL (2025-07-09 21:16:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 9 21:12:59.851276011 (396h2m34.621917346s) [upload=316866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:12:59.851280840 (396h2m34.621922175s) [upload=316866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:13:00.293986052 (396h2m35.064627387s) [seg_pos={0 0}, upload=316866]: Begin upload segment... Jul 9 21:13:00.294028513 (396h2m35.064669838s) [seg_pos={0 0}, upload=316866]: Done begin upload segment. Jul 9 21:13:00.294053310 (396h2m35.064694645s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294065965 (396h2m35.06470734s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294416946 (396h2m35.065058281s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294555371 (396h2m35.065196716s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294621748 (396h2m35.065263083s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294841869 (396h2m35.065483205s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294893679 (396h2m35.065535014s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295210985 (396h2m35.06585232s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295293353 (396h2m35.065934708s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295547890 (396h2m35.066189225s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295575663 (396h2m35.066216998s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295799371 (396h2m35.066440706s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295819099 (396h2m35.066460434s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295999824 (396h2m35.066641159s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296085999 (396h2m35.066727334s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296281614 (396h2m35.066922949s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296372067 (396h2m35.067013402s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296635730 (396h2m35.067277075s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:01.401192960 (396h2m36.171834295s) [upload=316866]: Next returned nil segment Jul 9 21:13:01.401213670 (396h2m36.171855025s) [upload=316866]: Waiting for error group managing segments... Jul 9 21:13:01.682850619 (396h2m36.453491954s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:01.684980433 (396h2m36.455621768s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.299147079 (396h2m37.069788414s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.302091481 (396h2m37.072732816s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.303576673 (396h2m37.074218008s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.325871773 (396h2m37.096513118s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.328765018 (396h2m37.099406363s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:03.478723068 (396h2m38.249364403s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:18.672666052 (396h2m53.443307397s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:23.418202017 (396h2m58.188843362s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:35.770329402 (396h3m10.540970737s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:59.211354079 (396h3m33.981995454s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211390067 (396h3m33.982031402s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12JT6NEDfWncJiSGaLgu12owQuPA8fGCv2cKa6z9kxS6TSLppha, address:104.36.82.36:4627): protocol: expected piece hash; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; EOF Jul 9 21:13:59.211457927 (396h3m33.982099253s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:104.36.82.60:4360): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211463268 (396h3m33.982104603s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12dQKfD74Hp3MiUYqMm2oxGfHiStWTXdg4c4MEq8gCZ1hcbkgaN, address:104.36.82.21:4484): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211557117 (396h3m33.982198452s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1km2oG8GFekofijwQYhJH65U5FZb2Qkji8YdNcjF3Xt3wCmx8b, address:45.91.95.2:4030): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211640486 (396h3m33.982281821s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1ovUMgZ6LCi9r3pmenbSbVDRcXx8X1JF29LLReoLkfXcWwnLbj, address:45.91.95.5:4007): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211689650 (396h3m33.982330985s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211715390 (396h3m33.982356725s) [seg_pos={0 0}, upload=316866]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 9 21:13:59.441970386 (396h3m34.212611771s) [upload=316866]: Done uploading segments... Jul 9 21:13:59.442001806 (396h3m34.212643161s) [upload=316866]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 45.095461744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X..X..........................] duration: 5.937318399s FAIL (2025-07-09 21:16:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.937342754s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X........X........X.X...XXX.........X......X....X..X.....] duration: 24.756651295s FAIL (2025-07-09 21:16:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.756673989s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..X.........................] duration: 5.942329991s FAIL (2025-07-09 21:15:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.942352472s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X..X..X..X..X..X....X..X..........X...X................X..] duration: 37.025428265s FAIL (2025-07-09 21:15:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 9 21:12:59.851276011 (396h2m34.621917346s) [upload=316866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:12:59.851280840 (396h2m34.621922175s) [upload=316866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:13:00.293986052 (396h2m35.064627387s) [seg_pos={0 0}, upload=316866]: Begin upload segment... Jul 9 21:13:00.294028513 (396h2m35.064669838s) [seg_pos={0 0}, upload=316866]: Done begin upload segment. Jul 9 21:13:00.294053310 (396h2m35.064694645s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294065965 (396h2m35.06470734s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294416946 (396h2m35.065058281s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294555371 (396h2m35.065196716s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294621748 (396h2m35.065263083s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294841869 (396h2m35.065483205s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294893679 (396h2m35.065535014s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295210985 (396h2m35.06585232s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295293353 (396h2m35.065934708s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295547890 (396h2m35.066189225s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295575663 (396h2m35.066216998s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295799371 (396h2m35.066440706s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295819099 (396h2m35.066460434s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295999824 (396h2m35.066641159s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296085999 (396h2m35.066727334s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296281614 (396h2m35.066922949s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296372067 (396h2m35.067013402s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296635730 (396h2m35.067277075s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:01.401192960 (396h2m36.171834295s) [upload=316866]: Next returned nil segment Jul 9 21:13:01.401213670 (396h2m36.171855025s) [upload=316866]: Waiting for error group managing segments... Jul 9 21:13:01.682850619 (396h2m36.453491954s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:01.684980433 (396h2m36.455621768s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.299147079 (396h2m37.069788414s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.302091481 (396h2m37.072732816s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.303576673 (396h2m37.074218008s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.325871773 (396h2m37.096513118s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.328765018 (396h2m37.099406363s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:03.478723068 (396h2m38.249364403s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:18.672666052 (396h2m53.443307397s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:23.418202017 (396h2m58.188843362s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:35.770329402 (396h3m10.540970737s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:59.211354079 (396h3m33.981995454s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211390067 (396h3m33.982031402s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12JT6NEDfWncJiSGaLgu12owQuPA8fGCv2cKa6z9kxS6TSLppha, address:104.36.82.36:4627): protocol: expected piece hash; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; EOF Jul 9 21:13:59.211457927 (396h3m33.982099253s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:104.36.82.60:4360): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211463268 (396h3m33.982104603s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12dQKfD74Hp3MiUYqMm2oxGfHiStWTXdg4c4MEq8gCZ1hcbkgaN, address:104.36.82.21:4484): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211557117 (396h3m33.982198452s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1km2oG8GFekofijwQYhJH65U5FZb2Qkji8YdNcjF3Xt3wCmx8b, address:45.91.95.2:4030): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211640486 (396h3m33.982281821s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1ovUMgZ6LCi9r3pmenbSbVDRcXx8X1JF29LLReoLkfXcWwnLbj, address:45.91.95.5:4007): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211689650 (396h3m33.982330985s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211715390 (396h3m33.982356725s) [seg_pos={0 0}, upload=316866]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 9 21:13:59.441970386 (396h3m34.212611771s) [upload=316866]: Done uploading segments... Jul 9 21:13:59.442001806 (396h3m34.212643161s) [upload=316866]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 37.025447662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [....X........X........X.X...XXX.........X......X....X..X....] duration: 20.347195242s FAIL (2025-07-09 21:15:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.347242572s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X..X........................] duration: 11.954541365s FAIL (2025-07-09 21:15:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.954607621s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X..X..X..X..X..X....X..X..........X...X................X.] duration: 45.140053201s FAIL (2025-07-09 21:14:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 9 21:12:59.851276011 (396h2m34.621917346s) [upload=316866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:12:59.851280840 (396h2m34.621922175s) [upload=316866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:13:00.293986052 (396h2m35.064627387s) [seg_pos={0 0}, upload=316866]: Begin upload segment... Jul 9 21:13:00.294028513 (396h2m35.064669838s) [seg_pos={0 0}, upload=316866]: Done begin upload segment. Jul 9 21:13:00.294053310 (396h2m35.064694645s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294065965 (396h2m35.06470734s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294416946 (396h2m35.065058281s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294555371 (396h2m35.065196716s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294621748 (396h2m35.065263083s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294841869 (396h2m35.065483205s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294893679 (396h2m35.065535014s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295210985 (396h2m35.06585232s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295293353 (396h2m35.065934708s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295547890 (396h2m35.066189225s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295575663 (396h2m35.066216998s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295799371 (396h2m35.066440706s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295819099 (396h2m35.066460434s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295999824 (396h2m35.066641159s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296085999 (396h2m35.066727334s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296281614 (396h2m35.066922949s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296372067 (396h2m35.067013402s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296635730 (396h2m35.067277075s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:01.401192960 (396h2m36.171834295s) [upload=316866]: Next returned nil segment Jul 9 21:13:01.401213670 (396h2m36.171855025s) [upload=316866]: Waiting for error group managing segments... Jul 9 21:13:01.682850619 (396h2m36.453491954s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:01.684980433 (396h2m36.455621768s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.299147079 (396h2m37.069788414s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.302091481 (396h2m37.072732816s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.303576673 (396h2m37.074218008s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.325871773 (396h2m37.096513118s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.328765018 (396h2m37.099406363s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:03.478723068 (396h2m38.249364403s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:18.672666052 (396h2m53.443307397s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:23.418202017 (396h2m58.188843362s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:35.770329402 (396h3m10.540970737s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:59.211354079 (396h3m33.981995454s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211390067 (396h3m33.982031402s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12JT6NEDfWncJiSGaLgu12owQuPA8fGCv2cKa6z9kxS6TSLppha, address:104.36.82.36:4627): protocol: expected piece hash; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; EOF Jul 9 21:13:59.211457927 (396h3m33.982099253s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:104.36.82.60:4360): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211463268 (396h3m33.982104603s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12dQKfD74Hp3MiUYqMm2oxGfHiStWTXdg4c4MEq8gCZ1hcbkgaN, address:104.36.82.21:4484): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211557117 (396h3m33.982198452s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1km2oG8GFekofijwQYhJH65U5FZb2Qkji8YdNcjF3Xt3wCmx8b, address:45.91.95.2:4030): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211640486 (396h3m33.982281821s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1ovUMgZ6LCi9r3pmenbSbVDRcXx8X1JF29LLReoLkfXcWwnLbj, address:45.91.95.5:4007): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211689650 (396h3m33.982330985s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211715390 (396h3m33.982356725s) [seg_pos={0 0}, upload=316866]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 9 21:13:59.441970386 (396h3m34.212611771s) [upload=316866]: Done uploading segments... Jul 9 21:13:59.442001806 (396h3m34.212643161s) [upload=316866]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 45.140079291s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.....X........X........X.X...XXX.........X......X....X..X...] duration: 43.456915497s FAIL (2025-07-09 21:14:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.456936316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X..X.......................] duration: 6.594687546s FAIL (2025-07-09 21:14:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.59471052s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X...X..X..X..X..X..X....X..X..........X...X................X] duration: 1m0.233356984s FAIL (2025-07-09 21:13:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 9 21:12:59.851276011 (396h2m34.621917346s) [upload=316866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:12:59.851280840 (396h2m34.621922175s) [upload=316866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:13:00.293986052 (396h2m35.064627387s) [seg_pos={0 0}, upload=316866]: Begin upload segment... Jul 9 21:13:00.294028513 (396h2m35.064669838s) [seg_pos={0 0}, upload=316866]: Done begin upload segment. Jul 9 21:13:00.294053310 (396h2m35.064694645s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294065965 (396h2m35.06470734s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294416946 (396h2m35.065058281s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294555371 (396h2m35.065196716s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294621748 (396h2m35.065263083s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294841869 (396h2m35.065483205s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.294893679 (396h2m35.065535014s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295210985 (396h2m35.06585232s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295293353 (396h2m35.065934708s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295547890 (396h2m35.066189225s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295575663 (396h2m35.066216998s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295799371 (396h2m35.066440706s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295819099 (396h2m35.066460434s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.295999824 (396h2m35.066641159s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296085999 (396h2m35.066727334s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296281614 (396h2m35.066922949s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296372067 (396h2m35.067013402s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:00.296635730 (396h2m35.067277075s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Uploading piece... Jul 9 21:13:01.401192960 (396h2m36.171834295s) [upload=316866]: Next returned nil segment Jul 9 21:13:01.401213670 (396h2m36.171855025s) [upload=316866]: Waiting for error group managing segments... Jul 9 21:13:01.682850619 (396h2m36.453491954s) [noise=true , address=79.127.205.235:4675 , piece_id=FIVJO24VQNIVBNTQPPYFGTDQOAGRHBKX35RIKVE7523IWGJ3DZZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:01.684980433 (396h2m36.455621768s) [noise=true , address=79.127.205.234:4601 , piece_id=4YJBLU2G7XBKIAO5XNRNA3FTMJMLOBG246IX7HF4ONBWDT3GL7KA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.299147079 (396h2m37.069788414s) [noise=true , address=77.90.31.18:4820 , piece_id=7D65UOMEEQSB22Q5322MIWFE3MRZOMDUEZ4PB4SZC4L5JI544CZQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.302091481 (396h2m37.072732816s) [noise=true , address=77.90.31.12:4624 , piece_id=PWB6LEXFXQIS3YNORQVHGF7B2HXQMQFM6Y2KQLEAUY7RJSLODKEQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.303576673 (396h2m37.074218008s) [noise=true , address=77.90.31.21:4894 , piece_id=KSHJGJ6LPVB6SJO3Z4PVOTCHKGAFDX6L55PGU5VQV2EKFNGUH3HQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.325871773 (396h2m37.096513118s) [noise=true , address=77.90.31.13:4634 , piece_id=4UD7PEWJJW7TCJICQFCWWCVVSVC52MF5N4PZMEJVTSL6UNSZ44GA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:02.328765018 (396h2m37.099406363s) [noise=true , address=77.90.31.11:4581 , piece_id=24EFD7L2XOFSWUZ2X3ILQXIEDGG2QD6EBNEFE3VM32OUYJ767MJQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:03.478723068 (396h2m38.249364403s) [noise=true , address=79.127.205.246:4263 , piece_id=MPEELS53EBRUNQXV6XXQQG5BOV2OMDR5FMJWX6CGHQSFQYHHVGVA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:18.672666052 (396h2m53.443307397s) [noise=true , address=104.36.82.34:4551 , piece_id=RUULCBAP4QKEOB67JQYNDWBER7YBX6STIS6262I4Q6MCYH3XOBXQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:23.418202017 (396h2m58.188843362s) [noise=true , address=104.36.82.23:4473 , piece_id=WQZMMESQ7XIKGMPIAVHTICR76XHN4QBFDS7CY3XELVWJQTDSVYLQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:35.770329402 (396h3m10.540970737s) [noise=true , address=45.91.95.4:4008 , piece_id=SDYETLXNDLNYMELPAOCNDVNHIX2XMB3HF7CGLYNIJ2FCBYUS2N4A, seg_pos={0 0}, upload=316866]: Done uploading piece. err: <nil> Jul 9 21:13:59.211354079 (396h3m33.981995454s) [noise=true , address=45.91.95.3:4021 , piece_id=6UVJZUNN4BEUQNT4D5VWHH3X4D4J6U3M6IWDA35VKPEJQYSXJARA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211390067 (396h3m33.982031402s) [noise=true , address=104.36.82.36:4627 , piece_id=657MWOU3ZZMKOFJ5YTW4PYMTW5YY3FNW5GOM4LF5QNVNULR2J7UA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12JT6NEDfWncJiSGaLgu12owQuPA8fGCv2cKa6z9kxS6TSLppha, address:104.36.82.36:4627): protocol: expected piece hash; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; write tcp 188.166.225.170:39228->104.36.82.36:4627: use of closed network connection; EOF Jul 9 21:13:59.211457927 (396h3m33.982099253s) [noise=true , address=104.36.82.60:4360 , piece_id=WBU7NBVHAZFG6H66CVXN6F7A2OKQLGVQ55QC7NUQA4NJRORWIOWQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:104.36.82.60:4360): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211463268 (396h3m33.982104603s) [noise=true , address=104.36.82.21:4484 , piece_id=4HZVUHWPBWJ6WXYXZCSLZOH57NPVHCSG2AZWX462CIGEISJ3MTMQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12dQKfD74Hp3MiUYqMm2oxGfHiStWTXdg4c4MEq8gCZ1hcbkgaN, address:104.36.82.21:4484): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211557117 (396h3m33.982198452s) [noise=true , address=45.91.95.2:4030 , piece_id=LWOF3CMMUB7SDZYDBNUIGNMSCAQDJRY3VOXFAZWILJI2LNXM34IQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1km2oG8GFekofijwQYhJH65U5FZb2Qkji8YdNcjF3Xt3wCmx8b, address:45.91.95.2:4030): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211640486 (396h3m33.982281821s) [noise=true , address=45.91.95.5:4007 , piece_id=ERTLLONTDLPMGRUB4UCEYWUZKT77SOH2ZCEHRCQ7POYVRWMMMROA, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:1ovUMgZ6LCi9r3pmenbSbVDRcXx8X1JF29LLReoLkfXcWwnLbj, address:45.91.95.5:4007): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211689650 (396h3m33.982330985s) [noise=true , address=104.36.82.19:4410 , piece_id=PQONN2YDRTO74CWDNX4Q5XOGGRGGWPZZXRLNFUIIACNYVDVZHWYQ, seg_pos={0 0}, upload=316866]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:13:59.211715390 (396h3m33.982356725s) [seg_pos={0 0}, upload=316866]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 9 21:13:59.441970386 (396h3m34.212611771s) [upload=316866]: Done uploading segments... Jul 9 21:13:59.442001806 (396h3m34.212643161s) [upload=316866]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.233371502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [......X........X........X.X...XXX.........X......X....X..X..] duration: 53.692158899s FAIL (2025-07-09 21:13:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.692192613s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X..X......................] duration: 26.858568563s FAIL (2025-07-09 21:13:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.85858811s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [XX...X..X..X..X..X..X....X..X..........X...X................] duration: 49.356289221s FAIL (2025-07-09 21:12:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 49.356304049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X..X.....................] duration: 22.186283774s FAIL (2025-07-09 21:12:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.186306737s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.XX...X..X..X..X..X..X....X..X..........X...X...............] duration: 47.656052347s FAIL (2025-07-09 21:12:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 47.656068138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.......X........X........X.X...XXX.........X......X....X..X.] duration: 28.08058882s FAIL (2025-07-09 21:12:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.080601042s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [........X........X........X.X...XXX.........X......X....X..X] duration: 1m0.001405856s FAIL (2025-07-09 21:11:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001424391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...................................X..X....................] duration: 5.889599749s FAIL (2025-07-09 21:11:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.889618445s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [..XX...X..X..X..X..X..X....X..X..........X...X..............] duration: 38.432438943s FAIL (2025-07-09 21:11:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 38.432457669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...XX...X..X..X..X..X..X....X..X..........X...X.............] duration: 44.236342456s FAIL (2025-07-09 21:10:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 44.236362405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...................................X..X...................] duration: 6.773686782s FAIL (2025-07-09 21:10:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.773708432s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.........X........X........X.X...XXX.........X......X....X..] duration: 37.06718364s FAIL (2025-07-09 21:10:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.067194991s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X.........X........X........X.X...XXX.........X......X....X.] duration: 52.220207945s FAIL (2025-07-09 21:10:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.220218415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [....XX...X..X..X..X..X..X....X..X..........X...X............] duration: 36.945344161s FAIL (2025-07-09 21:09:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 36.945365211s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...................................X..X..................] duration: 6.38419703s FAIL (2025-07-09 21:09:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.384227951s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.X.........X........X........X.X...XXX.........X......X....X] duration: 1m0.001021819s FAIL (2025-07-09 21:09:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001506586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.....XX...X..X..X..X..X..X....X..X..........X...X...........] duration: 18.654117539s FAIL (2025-07-09 21:08:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.65414488s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X...................................X..X.................] duration: 6.075663304s FAIL (2025-07-09 21:08:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.075679645s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [......XX...X..X..X..X..X..X....X..X..........X...X..........] duration: 36.190194396s FAIL (2025-07-09 21:08:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 36.190212661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X.X.........X........X........X.X...XXX.........X......X....] duration: 46.66953827s FAIL (2025-07-09 21:08:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.669550002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X...................................X..X................] duration: 5.933086782s FAIL (2025-07-09 21:07:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.933107573s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.......XX...X..X..X..X..X..X....X..X..........X...X.........] duration: 54.153943621s FAIL (2025-07-09 21:07:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 54.153967706s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X...................................X..X...............] duration: 5.94609465s FAIL (2025-07-09 21:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.946113997s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.X.X.........X........X........X.X...XXX.........X......X...] duration: 49.163778393s FAIL (2025-07-09 21:07:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.163795897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [........XX...X..X..X..X..X..X....X..X..........X...X........] duration: 6.278617935s FAIL (2025-07-09 21:06:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.278635628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.........XX...X..X..X..X..X..X....X..X..........X...X.......] duration: 53.088598716s FAIL (2025-07-09 21:06:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 53.088628113s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X.X.........X........X........X.X...XXX.........X......X..] duration: 43.877119997s FAIL (2025-07-09 21:06:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.877143181s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...................................X..X..............] duration: 6.428208686s FAIL (2025-07-09 21:06:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.428226179s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X.X.........X........X........X.X...XXX.........X......X.] duration: 24.117784389s FAIL (2025-07-09 21:05:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.117799047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X.........XX...X..X..X..X..X..X....X..X..........X...X......] duration: 30.648844155s FAIL (2025-07-09 21:05:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 30.648874465s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [....X.X.........X........X........X.X...XXX.........X......X] duration: 1m0.001316297s FAIL (2025-07-09 21:05:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001332568s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...................................X..X.............] duration: 6.155225184s FAIL (2025-07-09 21:05:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.155247869s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [XX.........XX...X..X..X..X..X..X....X..X..........X...X.....] duration: 53.499394296s FAIL (2025-07-09 21:04:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 53.499416118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X...................................X..X............] duration: 6.158075353s FAIL (2025-07-09 21:04:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.158099299s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.....X.X.........X........X........X.X...XXX.........X......] duration: 52.352046665s FAIL (2025-07-09 21:04:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.352068388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.XX.........XX...X..X..X..X..X..X....X..X..........X...X....] duration: 48.46467413s FAIL (2025-07-09 21:03:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 48.464697855s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X...................................X..X...........] duration: 6.2518432s FAIL (2025-07-09 21:03:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.251891954s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [......X.X.........X........X........X.X...XXX.........X.....] duration: 35.874880405s FAIL (2025-07-09 21:02:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.874901275s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)