Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.XX.........XX...X..X..X..X..X..X....X..X..........X...X....] duration: 48.46467413s FAIL (2025-07-09 21:03:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 48.464697855s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X...................................X..X...........] duration: 6.2518432s FAIL (2025-07-09 21:03:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.251891954s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [......X.X.........X........X........X.X...XXX.........X.....] duration: 35.874880405s FAIL (2025-07-09 21:02:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.874901275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..XX.........XX...X..X..X..X..X..X....X..X..........X...X...] duration: 20.677203099s FAIL (2025-07-09 21:02:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.677221254s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X...................................X..X..........] duration: 5.873236812s FAIL (2025-07-09 21:02:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.873257872s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...XX.........XX...X..X..X..X..X..X....X..X..........X...X..] duration: 50.149350301s FAIL (2025-07-09 21:02:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 50.149382132s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.......X.X.........X........X........X.X...XXX.........X....] duration: 28.617953963s FAIL (2025-07-09 21:01:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.617966957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [....XX.........XX...X..X..X..X..X..X....X..X..........X...X.] duration: 29.860372006s FAIL (2025-07-09 21:01:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.860391455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [........X.X.........X........X........X.X...XXX.........X...] duration: 54.72038521s FAIL (2025-07-09 21:00:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 54.720404639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X...................................X..X.........] duration: 5.922691811s FAIL (2025-07-09 21:00:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.922715144s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.....XX.........XX...X..X..X..X..X..X....X..X..........X...X] duration: 1m0.230869091s FAIL (2025-07-09 21:00:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 94]: Done starting upload Jul 9 20:59:46.374594269 (395h49m21.145235604s) [upload=316694]: Uploading segments... Jul 9 20:59:46.374747803 (395h49m21.145389148s) [upload=316694]: Got next segment. Inline: false Jul 9 20:59:46.374816905 (395h49m21.14545824s) [upload=316694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:59:46.374849787 (395h49m21.145491122s) [upload=316694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:59:46.816020600 (395h49m21.586661935s) [seg_pos={0 0}, upload=316694]: Begin upload segment... Jul 9 20:59:46.816090875 (395h49m21.58673221s) [seg_pos={0 0}, upload=316694]: Done begin upload segment. Jul 9 20:59:46.816112525 (395h49m21.58675386s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816153194 (395h49m21.586794519s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816471542 (395h49m21.587112877s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816672617 (395h49m21.587313953s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.816891837 (395h49m21.587533172s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.817262576 (395h49m21.587903911s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.818708852 (395h49m21.589350187s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819714096 (395h49m21.590355421s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.819946981 (395h49m21.590588316s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820227448 (395h49m21.590868773s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.820768333 (395h49m21.591409668s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821008642 (395h49m21.591649967s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821307654 (395h49m21.591948989s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821527986 (395h49m21.592169321s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.821772293 (395h49m21.592413618s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822026169 (395h49m21.592667504s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822269845 (395h49m21.59291118s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:46.822504495 (395h49m21.59314583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Uploading piece... Jul 9 20:59:48.405140601 (395h49m23.175781936s) [upload=316694]: Next returned nil segment Jul 9 20:59:48.405157944 (395h49m23.175799279s) [upload=316694]: Waiting for error group managing segments... Jul 9 20:59:48.811482230 (395h49m23.582123555s) [noise=true , address=77.90.31.15:4716 , piece_id=WYWPAZ47UMFC44TJN7OPXSCEEZGPOESETHBQG3GD7IGAKRFGFKCA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.823100808 (395h49m23.593742143s) [noise=true , address=77.90.31.11:4560 , piece_id=KUTHP33HJXK6JPWE7KGJOTY2GJLHNFNQ7TKPU3YVVYZU6BFYY6QA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.827114446 (395h49m23.597755781s) [noise=true , address=77.90.31.14:4678 , piece_id=OGYL6B7QYYVMQ4RTKRGADSVIT67PK73SMLNZ6BINCFJND5ROUM5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.830195371 (395h49m23.600836706s) [noise=true , address=77.90.31.17:4783 , piece_id=BXOJYR5VDJ7DFAOAFSQLN5AOF6KHBEQ2A3HZZ2U4UIKYVMPTHSMA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.834714066 (395h49m23.605355391s) [noise=true , address=77.90.31.21:4890 , piece_id=GGJMKBND7G264MYHIKLPJBXAECDNTZSOEFP7IYUDQXISBXPQFLSQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.840327425 (395h49m23.61096875s) [noise=true , address=77.90.31.19:4860 , piece_id=5KVMVN57RWFRB4S74GLIJWGHV67PRI5JGJVHX5T2VDXDQSB4JQBA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:48.858026095 (395h49m23.62866742s) [noise=true , address=77.90.31.12:4613 , piece_id=VYHFB4SXCXQ3GYGVJJEJREC4DIJQXVA4UOR54GSE5WKZPR2XU7QQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:49.013552248 (395h49m23.784193583s) [noise=true , address=79.127.219.47:4048 , piece_id=IG7YDXWZSHIQVDUR2YR3PLWYLXPRCSP46OIANHRZXGXQRG5MRRXA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 20:59:59.082416847 (395h49m33.853058202s) [noise=true , address=104.36.82.3:4775 , piece_id=CV6ZMJYMWRSVQAMALI54D7BCJY2MSQTPMQMWO3WAXR627IJSPQJQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:09.669829123 (395h49m44.440470468s) [noise=true , address=104.36.82.20:4440 , piece_id=3Q25DNLVVBS2YP7ETEPURYBEQZYHUHI27YSXDL47UBPWNRSYSPDA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:17.490099009 (395h49m52.260740344s) [noise=true , address=104.36.82.17:4360 , piece_id=X6BDLLDSTOVCH5BRY4HU4PUPC5BFMYN76I6KZUBZBOJ3DYS6TGJA, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:31.875475653 (395h50m6.646116978s) [noise=true , address=104.36.82.18:4372 , piece_id=GFQYMBMQFYQWY554UY66S4EBEDOZKX3CPKF2TPGGVCKILVE2ZQEQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:43.118905314 (395h50m17.889546679s) [noise=true , address=104.36.82.23:4450 , piece_id=E7XISJWMKKNNCRDLMP25MRD4RZJIIRH4RETC4H2U2B7MCOGGHGZQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: <nil> Jul 9 21:00:45.734838241 (395h50m20.505479577s) [noise=true , address=104.36.82.36:4632 , piece_id=WXLLQ6D7OWZ2PNZ5XFIB2ZA54PMVV5GDJPYUPKNYSVNVWKGULIMQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735154566 (395h50m20.505795901s) [noise=true , address=45.91.95.3:4022 , piece_id=UK5JDA3F65F4VBC3DVWKBMLTWIESJRNRA3PLJ54ZIN7FQRHJBPVQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, address:45.91.95.3:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735288052 (395h50m20.505929387s) [noise=true , address=45.91.95.5:4010 , piece_id=OAWREDXHU35VYQOV6TA3X4XGGVBK7DC3V376BM2ZKYUVVACWYY5A, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1R96dToDYH1ipG35JjiSWs3QaoUBtDnwNC8gLZ2Re7NLNFzvbp, address:45.91.95.5:4010): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735325092 (395h50m20.505966427s) [noise=true , address=45.91.95.4:4004 , piece_id=6NXSVYR4IUAOUMURR5KREGUUTXTTEM3N5AUFLIEX52YLT76Z7ZYQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:00:45.735360641 (395h50m20.506001976s) [noise=true , address=45.91.95.2:4008 , piece_id=GHVWOMDJQHFOKUARFBAMHVQOUJGRJYCXBNKOHOKI4RAP7M2SJRTQ, seg_pos={0 0}, upload=316694]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52094->45.91.95.2:4008: use of closed network connection Jul 9 21:00:45.735384036 (395h50m20.506025371s) [seg_pos={0 0}, upload=316694]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:00:45.964627290 (395h50m20.735268645s) [upload=316694]: Done uploading segments... Jul 9 21:00:45.964656335 (395h50m20.73529768s) [upload=316694]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.230894148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X...................................X..X........] duration: 6.452940565s FAIL (2025-07-09 21:00:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.452960774s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X........X.X.........X........X........X.X...XXX.........X..] duration: 24.264300092s FAIL (2025-07-09 20:59:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.264328365s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X...................................X..X.......] duration: 6.232771123s FAIL (2025-07-09 20:59:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.232793516s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [XX........X.X.........X........X........X.X...XXX.........X.] duration: 36.284966761s FAIL (2025-07-09 20:59:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.2849819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [......XX.........XX...X..X..X..X..X..X....X..X..........X...] duration: 31.389286884s FAIL (2025-07-09 20:59:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== oad=316661]: Got next segment. Inline: false Jul 9 20:57:09.481574744 (395h46m44.252216079s) [upload=316661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:57:09.481578972 (395h46m44.252220308s) [upload=316661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:57:09.925772958 (395h46m44.696414293s) [seg_pos={0 0}, upload=316661]: Begin upload segment... Jul 9 20:57:09.925846649 (395h46m44.696487984s) [seg_pos={0 0}, upload=316661]: Done begin upload segment. Jul 9 20:57:09.925869583 (395h46m44.696510908s) [noise=true , address=45.91.95.4:4012 , piece_id=IHTZ3AVW2K54AHPTCJTXSJDHKFQYRBJ6VMG3AWPMX4D6M7DSVWRQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.925962832 (395h46m44.696604177s) [noise=true , address=104.36.82.51:4056 , piece_id=E3AJ35JU34YDAHZI432AOAYSCC7TMIND2V4HLHIJTQG5LCTJXWRQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926206527 (395h46m44.696847862s) [noise=true , address=77.90.31.14:4687 , piece_id=4LH6JKFAD26N3ZWUWPJCJHHO75FV7THBPGTLZYXV62ETHXNDQ7PQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926455103 (395h46m44.697096428s) [noise=true , address=104.36.82.6:4880 , piece_id=PWT6SCZ4SGAQZD7NDQARHMVZNYDRHP5XV4U6E4LR5DMIUFWCF4CA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926559993 (395h46m44.697201328s) [noise=true , address=104.36.82.38:4708 , piece_id=LXSYAY3YTVT5Z2TWCGCQSZ372F5CP5L475DUI7LRCZLJFRADJJWQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926699851 (395h46m44.697341176s) [noise=true , address=45.91.95.2:4005 , piece_id=YKKJEORN4UUZWLCOO3666DZ2FNGPASNHRUMTYDNF2MIJOSBTWLLA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926938558 (395h46m44.697579883s) [noise=true , address=45.91.95.3:4001 , piece_id=W7P3HQEG4RQEAPO2KAXHHMUMSQVZN62II56FLWCLJNORCUW27YKQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926991950 (395h46m44.697633285s) [noise=true , address=104.36.82.4:4832 , piece_id=YX42JF7UFDKDX2QXP4UM5KI62Q5BSN6IESQCV5OR3MZ5OOGMB4GQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927159751 (395h46m44.697801086s) [noise=true , address=45.91.95.5:4002 , piece_id=EZ4RSL7IKDHQG2PLTJFR6N6YRQADFRNXVGUTLZYEWK5WJH3X65GA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927265413 (395h46m44.697906749s) [noise=true , address=104.36.82.40:4943 , piece_id=37XFBBMBJFKOEKIKSHSTA2CHMYK6I4Q54OOSX7IUL63R34IO5KFA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927423145 (395h46m44.69806447s) [noise=true , address=77.90.31.21:4898 , piece_id=FVYXUCWSLIYZAREVWV2G2Z27Y2WDRUQK45OV7WZVGPHW5RS3GV7Q, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927515362 (395h46m44.698156697s) [noise=true , address=77.90.31.19:4860 , piece_id=JA24EY5NBGW6IJUEALM3F45DQIXYXACFWAFELVOZCRNSHZXUJ5JQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927654448 (395h46m44.698295783s) [noise=true , address=79.127.219.38:4004 , piece_id=EN624ASKTAQ5RYXBLQZW5QAHMBBQPVTUSAJOGQVQKR4FX4GEBUIQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927753647 (395h46m44.698394982s) [noise=true , address=77.90.31.11:4557 , piece_id=SG6TAPMMYIAJFWKK4HX77T3UNKP6DAGMMS72ZIMZL6LFV7CLYKGQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927897242 (395h46m44.698538567s) [noise=true , address=104.36.82.22:4513 , piece_id=CPDA5GKOYFVIBQCO3BEWZ7RXH56JKDU64UUR4LDODRHOI7GBKBLQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927990350 (395h46m44.698631685s) [noise=true , address=77.90.31.19:4853 , piece_id=EBGKM4AN2HM376IQVGJI4HBXYV4SDU2DCOIEKHATCMNSJ44RSHEQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.928216142 (395h46m44.698857467s) [noise=true , address=77.90.31.16:4747 , piece_id=66IKFVQC5RZYEOSIH6KK4AMZPIP2POZOCESVLCYWCYWOJ3KHSWXA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.928439940 (395h46m44.699081275s) [noise=true , address=104.36.82.7:4927 , piece_id=CYPSRZUD7YT5ZV7FUXQOCNZW4Y7S7VIRELQT7IEL66NRQXESECCA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:11.491476539 (395h46m46.262117864s) [upload=316661]: Next returned nil segment Jul 9 20:57:11.491493962 (395h46m46.262135288s) [upload=316661]: Waiting for error group managing segments... Jul 9 20:57:11.913920504 (395h46m46.684561859s) [noise=true , address=77.90.31.21:4898 , piece_id=FVYXUCWSLIYZAREVWV2G2Z27Y2WDRUQK45OV7WZVGPHW5RS3GV7Q, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.914410241 (395h46m46.685051576s) [noise=true , address=77.90.31.11:4557 , piece_id=SG6TAPMMYIAJFWKK4HX77T3UNKP6DAGMMS72ZIMZL6LFV7CLYKGQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.914998206 (395h46m46.685639541s) [noise=true , address=79.127.219.38:4004 , piece_id=EN624ASKTAQ5RYXBLQZW5QAHMBBQPVTUSAJOGQVQKR4FX4GEBUIQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.929405148 (395h46m46.700046483s) [noise=true , address=77.90.31.14:4687 , piece_id=4LH6JKFAD26N3ZWUWPJCJHHO75FV7THBPGTLZYXV62ETHXNDQ7PQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.932087910 (395h46m46.702729245s) [noise=true , address=77.90.31.16:4747 , piece_id=66IKFVQC5RZYEOSIH6KK4AMZPIP2POZOCESVLCYWCYWOJ3KHSWXA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.937636836 (395h46m46.708278161s) [noise=true , address=77.90.31.19:4853 , piece_id=EBGKM4AN2HM376IQVGJI4HBXYV4SDU2DCOIEKHATCMNSJ44RSHEQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.939939832 (395h46m46.710581167s) [noise=true , address=77.90.31.19:4860 , piece_id=JA24EY5NBGW6IJUEALM3F45DQIXYXACFWAFELVOZCRNSHZXUJ5JQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:24.277638949 (395h46m59.048280284s) [noise=true , address=45.91.95.4:4012 , piece_id=IHTZ3AVW2K54AHPTCJTXSJDHKFQYRBJ6VMG3AWPMX4D6M7DSVWRQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:12kFYkJi2nr3EVpxZ6NVWcQwX8hRy3fXkfVmBNXjcmyPK2EoHMt, address:45.91.95.4:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:57:33.774824721 (395h47m8.545466046s) [noise=true , address=104.36.82.4:4832 , piece_id=YX42JF7UFDKDX2QXP4UM5KI62Q5BSN6IESQCV5OR3MZ5OOGMB4GQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:42.554846121 (395h47m17.325487456s) [noise=true , address=104.36.82.38:4708 , piece_id=LXSYAY3YTVT5Z2TWCGCQSZ372F5CP5L475DUI7LRCZLJFRADJJWQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:43.625027569 (395h47m18.395668904s) [noise=true , address=104.36.82.6:4880 , piece_id=PWT6SCZ4SGAQZD7NDQARHMVZNYDRHP5XV4U6E4LR5DMIUFWCF4CA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:51.567654956 (395h47m26.338296281s) [noise=true , address=104.36.82.40:4943 , piece_id=37XFBBMBJFKOEKIKSHSTA2CHMYK6I4Q54OOSX7IUL63R34IO5KFA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:52.866642294 (395h47m27.637283629s) [noise=true , address=104.36.82.7:4927 , piece_id=CYPSRZUD7YT5ZV7FUXQOCNZW4Y7S7VIRELQT7IEL66NRQXESECCA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:58:08.840679629 (395h47m43.611320964s) [noise=true , address=45.91.95.5:4002 , piece_id=EZ4RSL7IKDHQG2PLTJFR6N6YRQADFRNXVGUTLZYEWK5WJH3X65GA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:16MgCJPC1CuwnutbjFMaHYn7fmDzDoP7wyFYChw6m1WDKMLz4x, address:45.91.95.5:4002): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840712843 (395h47m43.611354168s) [noise=true , address=104.36.82.22:4513 , piece_id=CPDA5GKOYFVIBQCO3BEWZ7RXH56JKDU64UUR4LDODRHOI7GBKBLQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:104.36.82.22:4513): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840728242 (395h47m43.611369577s) [noise=true , address=104.36.82.51:4056 , piece_id=E3AJ35JU34YDAHZI432AOAYSCC7TMIND2V4HLHIJTQG5LCTJXWRQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1nY93g6WbZs2rfBYoKtf4Z7xGSr1QwizrBibKjpgKFbTbnVe1c, address:104.36.82.51:4056): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840744012 (395h47m43.611385347s) [noise=true , address=45.91.95.2:4005 , piece_id=YKKJEORN4UUZWLCOO3666DZ2FNGPASNHRUMTYDNF2MIJOSBTWLLA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1bxFpBmz99gV4rL6bMtnHuthK2Xg4EusbfMAWF37RTRtPkqhQS, address:45.91.95.2:4005): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840749212 (395h47m43.611390547s) [noise=true , address=45.91.95.3:4001 , piece_id=W7P3HQEG4RQEAPO2KAXHHMUMSQVZN62II56FLWCLJNORCUW27YKQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1L6dYFtTZvGGF9XsUFDHBBaSNPTnYjUVPT7VU5hfVpJpmpn3jL, address:45.91.95.3:4001): protocol: write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection; write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection Jul 9 20:58:08.840772637 (395h47m43.611413972s) [seg_pos={0 0}, upload=316661]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 20:58:09.067780024 (395h47m43.838421389s) [upload=316661]: Done uploading segments... Jul 9 20:58:09.067830080 (395h47m43.838471415s) [upload=316661]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 31.389336339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X...................................X..X......] duration: 6.217725626s FAIL (2025-07-09 20:59:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.21774887s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X...................................X..X.....] duration: 8.746570717s FAIL (2025-07-09 20:59:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.746593732s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.......XX.........XX...X..X..X..X..X..X....X..X..........X..] duration: 23.793911147s FAIL (2025-07-09 20:58:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== oad=316661]: Got next segment. Inline: false Jul 9 20:57:09.481574744 (395h46m44.252216079s) [upload=316661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:57:09.481578972 (395h46m44.252220308s) [upload=316661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:57:09.925772958 (395h46m44.696414293s) [seg_pos={0 0}, upload=316661]: Begin upload segment... Jul 9 20:57:09.925846649 (395h46m44.696487984s) [seg_pos={0 0}, upload=316661]: Done begin upload segment. Jul 9 20:57:09.925869583 (395h46m44.696510908s) [noise=true , address=45.91.95.4:4012 , piece_id=IHTZ3AVW2K54AHPTCJTXSJDHKFQYRBJ6VMG3AWPMX4D6M7DSVWRQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.925962832 (395h46m44.696604177s) [noise=true , address=104.36.82.51:4056 , piece_id=E3AJ35JU34YDAHZI432AOAYSCC7TMIND2V4HLHIJTQG5LCTJXWRQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926206527 (395h46m44.696847862s) [noise=true , address=77.90.31.14:4687 , piece_id=4LH6JKFAD26N3ZWUWPJCJHHO75FV7THBPGTLZYXV62ETHXNDQ7PQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926455103 (395h46m44.697096428s) [noise=true , address=104.36.82.6:4880 , piece_id=PWT6SCZ4SGAQZD7NDQARHMVZNYDRHP5XV4U6E4LR5DMIUFWCF4CA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926559993 (395h46m44.697201328s) [noise=true , address=104.36.82.38:4708 , piece_id=LXSYAY3YTVT5Z2TWCGCQSZ372F5CP5L475DUI7LRCZLJFRADJJWQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926699851 (395h46m44.697341176s) [noise=true , address=45.91.95.2:4005 , piece_id=YKKJEORN4UUZWLCOO3666DZ2FNGPASNHRUMTYDNF2MIJOSBTWLLA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926938558 (395h46m44.697579883s) [noise=true , address=45.91.95.3:4001 , piece_id=W7P3HQEG4RQEAPO2KAXHHMUMSQVZN62II56FLWCLJNORCUW27YKQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926991950 (395h46m44.697633285s) [noise=true , address=104.36.82.4:4832 , piece_id=YX42JF7UFDKDX2QXP4UM5KI62Q5BSN6IESQCV5OR3MZ5OOGMB4GQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927159751 (395h46m44.697801086s) [noise=true , address=45.91.95.5:4002 , piece_id=EZ4RSL7IKDHQG2PLTJFR6N6YRQADFRNXVGUTLZYEWK5WJH3X65GA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927265413 (395h46m44.697906749s) [noise=true , address=104.36.82.40:4943 , piece_id=37XFBBMBJFKOEKIKSHSTA2CHMYK6I4Q54OOSX7IUL63R34IO5KFA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927423145 (395h46m44.69806447s) [noise=true , address=77.90.31.21:4898 , piece_id=FVYXUCWSLIYZAREVWV2G2Z27Y2WDRUQK45OV7WZVGPHW5RS3GV7Q, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927515362 (395h46m44.698156697s) [noise=true , address=77.90.31.19:4860 , piece_id=JA24EY5NBGW6IJUEALM3F45DQIXYXACFWAFELVOZCRNSHZXUJ5JQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927654448 (395h46m44.698295783s) [noise=true , address=79.127.219.38:4004 , piece_id=EN624ASKTAQ5RYXBLQZW5QAHMBBQPVTUSAJOGQVQKR4FX4GEBUIQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927753647 (395h46m44.698394982s) [noise=true , address=77.90.31.11:4557 , piece_id=SG6TAPMMYIAJFWKK4HX77T3UNKP6DAGMMS72ZIMZL6LFV7CLYKGQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927897242 (395h46m44.698538567s) [noise=true , address=104.36.82.22:4513 , piece_id=CPDA5GKOYFVIBQCO3BEWZ7RXH56JKDU64UUR4LDODRHOI7GBKBLQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927990350 (395h46m44.698631685s) [noise=true , address=77.90.31.19:4853 , piece_id=EBGKM4AN2HM376IQVGJI4HBXYV4SDU2DCOIEKHATCMNSJ44RSHEQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.928216142 (395h46m44.698857467s) [noise=true , address=77.90.31.16:4747 , piece_id=66IKFVQC5RZYEOSIH6KK4AMZPIP2POZOCESVLCYWCYWOJ3KHSWXA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.928439940 (395h46m44.699081275s) [noise=true , address=104.36.82.7:4927 , piece_id=CYPSRZUD7YT5ZV7FUXQOCNZW4Y7S7VIRELQT7IEL66NRQXESECCA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:11.491476539 (395h46m46.262117864s) [upload=316661]: Next returned nil segment Jul 9 20:57:11.491493962 (395h46m46.262135288s) [upload=316661]: Waiting for error group managing segments... Jul 9 20:57:11.913920504 (395h46m46.684561859s) [noise=true , address=77.90.31.21:4898 , piece_id=FVYXUCWSLIYZAREVWV2G2Z27Y2WDRUQK45OV7WZVGPHW5RS3GV7Q, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.914410241 (395h46m46.685051576s) [noise=true , address=77.90.31.11:4557 , piece_id=SG6TAPMMYIAJFWKK4HX77T3UNKP6DAGMMS72ZIMZL6LFV7CLYKGQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.914998206 (395h46m46.685639541s) [noise=true , address=79.127.219.38:4004 , piece_id=EN624ASKTAQ5RYXBLQZW5QAHMBBQPVTUSAJOGQVQKR4FX4GEBUIQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.929405148 (395h46m46.700046483s) [noise=true , address=77.90.31.14:4687 , piece_id=4LH6JKFAD26N3ZWUWPJCJHHO75FV7THBPGTLZYXV62ETHXNDQ7PQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.932087910 (395h46m46.702729245s) [noise=true , address=77.90.31.16:4747 , piece_id=66IKFVQC5RZYEOSIH6KK4AMZPIP2POZOCESVLCYWCYWOJ3KHSWXA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.937636836 (395h46m46.708278161s) [noise=true , address=77.90.31.19:4853 , piece_id=EBGKM4AN2HM376IQVGJI4HBXYV4SDU2DCOIEKHATCMNSJ44RSHEQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.939939832 (395h46m46.710581167s) [noise=true , address=77.90.31.19:4860 , piece_id=JA24EY5NBGW6IJUEALM3F45DQIXYXACFWAFELVOZCRNSHZXUJ5JQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:24.277638949 (395h46m59.048280284s) [noise=true , address=45.91.95.4:4012 , piece_id=IHTZ3AVW2K54AHPTCJTXSJDHKFQYRBJ6VMG3AWPMX4D6M7DSVWRQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:12kFYkJi2nr3EVpxZ6NVWcQwX8hRy3fXkfVmBNXjcmyPK2EoHMt, address:45.91.95.4:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:57:33.774824721 (395h47m8.545466046s) [noise=true , address=104.36.82.4:4832 , piece_id=YX42JF7UFDKDX2QXP4UM5KI62Q5BSN6IESQCV5OR3MZ5OOGMB4GQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:42.554846121 (395h47m17.325487456s) [noise=true , address=104.36.82.38:4708 , piece_id=LXSYAY3YTVT5Z2TWCGCQSZ372F5CP5L475DUI7LRCZLJFRADJJWQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:43.625027569 (395h47m18.395668904s) [noise=true , address=104.36.82.6:4880 , piece_id=PWT6SCZ4SGAQZD7NDQARHMVZNYDRHP5XV4U6E4LR5DMIUFWCF4CA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:51.567654956 (395h47m26.338296281s) [noise=true , address=104.36.82.40:4943 , piece_id=37XFBBMBJFKOEKIKSHSTA2CHMYK6I4Q54OOSX7IUL63R34IO5KFA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:52.866642294 (395h47m27.637283629s) [noise=true , address=104.36.82.7:4927 , piece_id=CYPSRZUD7YT5ZV7FUXQOCNZW4Y7S7VIRELQT7IEL66NRQXESECCA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:58:08.840679629 (395h47m43.611320964s) [noise=true , address=45.91.95.5:4002 , piece_id=EZ4RSL7IKDHQG2PLTJFR6N6YRQADFRNXVGUTLZYEWK5WJH3X65GA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:16MgCJPC1CuwnutbjFMaHYn7fmDzDoP7wyFYChw6m1WDKMLz4x, address:45.91.95.5:4002): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840712843 (395h47m43.611354168s) [noise=true , address=104.36.82.22:4513 , piece_id=CPDA5GKOYFVIBQCO3BEWZ7RXH56JKDU64UUR4LDODRHOI7GBKBLQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:104.36.82.22:4513): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840728242 (395h47m43.611369577s) [noise=true , address=104.36.82.51:4056 , piece_id=E3AJ35JU34YDAHZI432AOAYSCC7TMIND2V4HLHIJTQG5LCTJXWRQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1nY93g6WbZs2rfBYoKtf4Z7xGSr1QwizrBibKjpgKFbTbnVe1c, address:104.36.82.51:4056): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840744012 (395h47m43.611385347s) [noise=true , address=45.91.95.2:4005 , piece_id=YKKJEORN4UUZWLCOO3666DZ2FNGPASNHRUMTYDNF2MIJOSBTWLLA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1bxFpBmz99gV4rL6bMtnHuthK2Xg4EusbfMAWF37RTRtPkqhQS, address:45.91.95.2:4005): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840749212 (395h47m43.611390547s) [noise=true , address=45.91.95.3:4001 , piece_id=W7P3HQEG4RQEAPO2KAXHHMUMSQVZN62II56FLWCLJNORCUW27YKQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1L6dYFtTZvGGF9XsUFDHBBaSNPTnYjUVPT7VU5hfVpJpmpn3jL, address:45.91.95.3:4001): protocol: write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection; write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection Jul 9 20:58:08.840772637 (395h47m43.611413972s) [seg_pos={0 0}, upload=316661]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 20:58:09.067780024 (395h47m43.838421389s) [upload=316661]: Done uploading segments... Jul 9 20:58:09.067830080 (395h47m43.838471415s) [upload=316661]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 23.793930303s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X...................................X..X....] duration: 11.42256101s FAIL (2025-07-09 20:58:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.422589054s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.XX........X.X.........X........X........X.X...XXX.........X] duration: 1m0.00121182s FAIL (2025-07-09 20:58:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001222561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [........XX.........XX...X..X..X..X..X..X....X..X..........X.] duration: 25.199862853s FAIL (2025-07-09 20:58:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== oad=316661]: Got next segment. Inline: false Jul 9 20:57:09.481574744 (395h46m44.252216079s) [upload=316661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:57:09.481578972 (395h46m44.252220308s) [upload=316661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:57:09.925772958 (395h46m44.696414293s) [seg_pos={0 0}, upload=316661]: Begin upload segment... Jul 9 20:57:09.925846649 (395h46m44.696487984s) [seg_pos={0 0}, upload=316661]: Done begin upload segment. Jul 9 20:57:09.925869583 (395h46m44.696510908s) [noise=true , address=45.91.95.4:4012 , piece_id=IHTZ3AVW2K54AHPTCJTXSJDHKFQYRBJ6VMG3AWPMX4D6M7DSVWRQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.925962832 (395h46m44.696604177s) [noise=true , address=104.36.82.51:4056 , piece_id=E3AJ35JU34YDAHZI432AOAYSCC7TMIND2V4HLHIJTQG5LCTJXWRQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926206527 (395h46m44.696847862s) [noise=true , address=77.90.31.14:4687 , piece_id=4LH6JKFAD26N3ZWUWPJCJHHO75FV7THBPGTLZYXV62ETHXNDQ7PQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926455103 (395h46m44.697096428s) [noise=true , address=104.36.82.6:4880 , piece_id=PWT6SCZ4SGAQZD7NDQARHMVZNYDRHP5XV4U6E4LR5DMIUFWCF4CA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926559993 (395h46m44.697201328s) [noise=true , address=104.36.82.38:4708 , piece_id=LXSYAY3YTVT5Z2TWCGCQSZ372F5CP5L475DUI7LRCZLJFRADJJWQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926699851 (395h46m44.697341176s) [noise=true , address=45.91.95.2:4005 , piece_id=YKKJEORN4UUZWLCOO3666DZ2FNGPASNHRUMTYDNF2MIJOSBTWLLA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926938558 (395h46m44.697579883s) [noise=true , address=45.91.95.3:4001 , piece_id=W7P3HQEG4RQEAPO2KAXHHMUMSQVZN62II56FLWCLJNORCUW27YKQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926991950 (395h46m44.697633285s) [noise=true , address=104.36.82.4:4832 , piece_id=YX42JF7UFDKDX2QXP4UM5KI62Q5BSN6IESQCV5OR3MZ5OOGMB4GQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927159751 (395h46m44.697801086s) [noise=true , address=45.91.95.5:4002 , piece_id=EZ4RSL7IKDHQG2PLTJFR6N6YRQADFRNXVGUTLZYEWK5WJH3X65GA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927265413 (395h46m44.697906749s) [noise=true , address=104.36.82.40:4943 , piece_id=37XFBBMBJFKOEKIKSHSTA2CHMYK6I4Q54OOSX7IUL63R34IO5KFA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927423145 (395h46m44.69806447s) [noise=true , address=77.90.31.21:4898 , piece_id=FVYXUCWSLIYZAREVWV2G2Z27Y2WDRUQK45OV7WZVGPHW5RS3GV7Q, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927515362 (395h46m44.698156697s) [noise=true , address=77.90.31.19:4860 , piece_id=JA24EY5NBGW6IJUEALM3F45DQIXYXACFWAFELVOZCRNSHZXUJ5JQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927654448 (395h46m44.698295783s) [noise=true , address=79.127.219.38:4004 , piece_id=EN624ASKTAQ5RYXBLQZW5QAHMBBQPVTUSAJOGQVQKR4FX4GEBUIQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927753647 (395h46m44.698394982s) [noise=true , address=77.90.31.11:4557 , piece_id=SG6TAPMMYIAJFWKK4HX77T3UNKP6DAGMMS72ZIMZL6LFV7CLYKGQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927897242 (395h46m44.698538567s) [noise=true , address=104.36.82.22:4513 , piece_id=CPDA5GKOYFVIBQCO3BEWZ7RXH56JKDU64UUR4LDODRHOI7GBKBLQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927990350 (395h46m44.698631685s) [noise=true , address=77.90.31.19:4853 , piece_id=EBGKM4AN2HM376IQVGJI4HBXYV4SDU2DCOIEKHATCMNSJ44RSHEQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.928216142 (395h46m44.698857467s) [noise=true , address=77.90.31.16:4747 , piece_id=66IKFVQC5RZYEOSIH6KK4AMZPIP2POZOCESVLCYWCYWOJ3KHSWXA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.928439940 (395h46m44.699081275s) [noise=true , address=104.36.82.7:4927 , piece_id=CYPSRZUD7YT5ZV7FUXQOCNZW4Y7S7VIRELQT7IEL66NRQXESECCA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:11.491476539 (395h46m46.262117864s) [upload=316661]: Next returned nil segment Jul 9 20:57:11.491493962 (395h46m46.262135288s) [upload=316661]: Waiting for error group managing segments... Jul 9 20:57:11.913920504 (395h46m46.684561859s) [noise=true , address=77.90.31.21:4898 , piece_id=FVYXUCWSLIYZAREVWV2G2Z27Y2WDRUQK45OV7WZVGPHW5RS3GV7Q, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.914410241 (395h46m46.685051576s) [noise=true , address=77.90.31.11:4557 , piece_id=SG6TAPMMYIAJFWKK4HX77T3UNKP6DAGMMS72ZIMZL6LFV7CLYKGQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.914998206 (395h46m46.685639541s) [noise=true , address=79.127.219.38:4004 , piece_id=EN624ASKTAQ5RYXBLQZW5QAHMBBQPVTUSAJOGQVQKR4FX4GEBUIQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.929405148 (395h46m46.700046483s) [noise=true , address=77.90.31.14:4687 , piece_id=4LH6JKFAD26N3ZWUWPJCJHHO75FV7THBPGTLZYXV62ETHXNDQ7PQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.932087910 (395h46m46.702729245s) [noise=true , address=77.90.31.16:4747 , piece_id=66IKFVQC5RZYEOSIH6KK4AMZPIP2POZOCESVLCYWCYWOJ3KHSWXA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.937636836 (395h46m46.708278161s) [noise=true , address=77.90.31.19:4853 , piece_id=EBGKM4AN2HM376IQVGJI4HBXYV4SDU2DCOIEKHATCMNSJ44RSHEQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.939939832 (395h46m46.710581167s) [noise=true , address=77.90.31.19:4860 , piece_id=JA24EY5NBGW6IJUEALM3F45DQIXYXACFWAFELVOZCRNSHZXUJ5JQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:24.277638949 (395h46m59.048280284s) [noise=true , address=45.91.95.4:4012 , piece_id=IHTZ3AVW2K54AHPTCJTXSJDHKFQYRBJ6VMG3AWPMX4D6M7DSVWRQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:12kFYkJi2nr3EVpxZ6NVWcQwX8hRy3fXkfVmBNXjcmyPK2EoHMt, address:45.91.95.4:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:57:33.774824721 (395h47m8.545466046s) [noise=true , address=104.36.82.4:4832 , piece_id=YX42JF7UFDKDX2QXP4UM5KI62Q5BSN6IESQCV5OR3MZ5OOGMB4GQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:42.554846121 (395h47m17.325487456s) [noise=true , address=104.36.82.38:4708 , piece_id=LXSYAY3YTVT5Z2TWCGCQSZ372F5CP5L475DUI7LRCZLJFRADJJWQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:43.625027569 (395h47m18.395668904s) [noise=true , address=104.36.82.6:4880 , piece_id=PWT6SCZ4SGAQZD7NDQARHMVZNYDRHP5XV4U6E4LR5DMIUFWCF4CA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:51.567654956 (395h47m26.338296281s) [noise=true , address=104.36.82.40:4943 , piece_id=37XFBBMBJFKOEKIKSHSTA2CHMYK6I4Q54OOSX7IUL63R34IO5KFA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:52.866642294 (395h47m27.637283629s) [noise=true , address=104.36.82.7:4927 , piece_id=CYPSRZUD7YT5ZV7FUXQOCNZW4Y7S7VIRELQT7IEL66NRQXESECCA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:58:08.840679629 (395h47m43.611320964s) [noise=true , address=45.91.95.5:4002 , piece_id=EZ4RSL7IKDHQG2PLTJFR6N6YRQADFRNXVGUTLZYEWK5WJH3X65GA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:16MgCJPC1CuwnutbjFMaHYn7fmDzDoP7wyFYChw6m1WDKMLz4x, address:45.91.95.5:4002): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840712843 (395h47m43.611354168s) [noise=true , address=104.36.82.22:4513 , piece_id=CPDA5GKOYFVIBQCO3BEWZ7RXH56JKDU64UUR4LDODRHOI7GBKBLQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:104.36.82.22:4513): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840728242 (395h47m43.611369577s) [noise=true , address=104.36.82.51:4056 , piece_id=E3AJ35JU34YDAHZI432AOAYSCC7TMIND2V4HLHIJTQG5LCTJXWRQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1nY93g6WbZs2rfBYoKtf4Z7xGSr1QwizrBibKjpgKFbTbnVe1c, address:104.36.82.51:4056): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840744012 (395h47m43.611385347s) [noise=true , address=45.91.95.2:4005 , piece_id=YKKJEORN4UUZWLCOO3666DZ2FNGPASNHRUMTYDNF2MIJOSBTWLLA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1bxFpBmz99gV4rL6bMtnHuthK2Xg4EusbfMAWF37RTRtPkqhQS, address:45.91.95.2:4005): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840749212 (395h47m43.611390547s) [noise=true , address=45.91.95.3:4001 , piece_id=W7P3HQEG4RQEAPO2KAXHHMUMSQVZN62II56FLWCLJNORCUW27YKQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1L6dYFtTZvGGF9XsUFDHBBaSNPTnYjUVPT7VU5hfVpJpmpn3jL, address:45.91.95.3:4001): protocol: write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection; write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection Jul 9 20:58:08.840772637 (395h47m43.611413972s) [seg_pos={0 0}, upload=316661]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 20:58:09.067780024 (395h47m43.838421389s) [upload=316661]: Done uploading segments... Jul 9 20:58:09.067830080 (395h47m43.838471415s) [upload=316661]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 25.199883391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.........XX.........XX...X..X..X..X..X..X....X..X..........X] duration: 1m0.228545918s FAIL (2025-07-09 20:58:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== oad=316661]: Got next segment. Inline: false Jul 9 20:57:09.481574744 (395h46m44.252216079s) [upload=316661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:57:09.481578972 (395h46m44.252220308s) [upload=316661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:57:09.925772958 (395h46m44.696414293s) [seg_pos={0 0}, upload=316661]: Begin upload segment... Jul 9 20:57:09.925846649 (395h46m44.696487984s) [seg_pos={0 0}, upload=316661]: Done begin upload segment. Jul 9 20:57:09.925869583 (395h46m44.696510908s) [noise=true , address=45.91.95.4:4012 , piece_id=IHTZ3AVW2K54AHPTCJTXSJDHKFQYRBJ6VMG3AWPMX4D6M7DSVWRQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.925962832 (395h46m44.696604177s) [noise=true , address=104.36.82.51:4056 , piece_id=E3AJ35JU34YDAHZI432AOAYSCC7TMIND2V4HLHIJTQG5LCTJXWRQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926206527 (395h46m44.696847862s) [noise=true , address=77.90.31.14:4687 , piece_id=4LH6JKFAD26N3ZWUWPJCJHHO75FV7THBPGTLZYXV62ETHXNDQ7PQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926455103 (395h46m44.697096428s) [noise=true , address=104.36.82.6:4880 , piece_id=PWT6SCZ4SGAQZD7NDQARHMVZNYDRHP5XV4U6E4LR5DMIUFWCF4CA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926559993 (395h46m44.697201328s) [noise=true , address=104.36.82.38:4708 , piece_id=LXSYAY3YTVT5Z2TWCGCQSZ372F5CP5L475DUI7LRCZLJFRADJJWQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926699851 (395h46m44.697341176s) [noise=true , address=45.91.95.2:4005 , piece_id=YKKJEORN4UUZWLCOO3666DZ2FNGPASNHRUMTYDNF2MIJOSBTWLLA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926938558 (395h46m44.697579883s) [noise=true , address=45.91.95.3:4001 , piece_id=W7P3HQEG4RQEAPO2KAXHHMUMSQVZN62II56FLWCLJNORCUW27YKQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.926991950 (395h46m44.697633285s) [noise=true , address=104.36.82.4:4832 , piece_id=YX42JF7UFDKDX2QXP4UM5KI62Q5BSN6IESQCV5OR3MZ5OOGMB4GQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927159751 (395h46m44.697801086s) [noise=true , address=45.91.95.5:4002 , piece_id=EZ4RSL7IKDHQG2PLTJFR6N6YRQADFRNXVGUTLZYEWK5WJH3X65GA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927265413 (395h46m44.697906749s) [noise=true , address=104.36.82.40:4943 , piece_id=37XFBBMBJFKOEKIKSHSTA2CHMYK6I4Q54OOSX7IUL63R34IO5KFA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927423145 (395h46m44.69806447s) [noise=true , address=77.90.31.21:4898 , piece_id=FVYXUCWSLIYZAREVWV2G2Z27Y2WDRUQK45OV7WZVGPHW5RS3GV7Q, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927515362 (395h46m44.698156697s) [noise=true , address=77.90.31.19:4860 , piece_id=JA24EY5NBGW6IJUEALM3F45DQIXYXACFWAFELVOZCRNSHZXUJ5JQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927654448 (395h46m44.698295783s) [noise=true , address=79.127.219.38:4004 , piece_id=EN624ASKTAQ5RYXBLQZW5QAHMBBQPVTUSAJOGQVQKR4FX4GEBUIQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927753647 (395h46m44.698394982s) [noise=true , address=77.90.31.11:4557 , piece_id=SG6TAPMMYIAJFWKK4HX77T3UNKP6DAGMMS72ZIMZL6LFV7CLYKGQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927897242 (395h46m44.698538567s) [noise=true , address=104.36.82.22:4513 , piece_id=CPDA5GKOYFVIBQCO3BEWZ7RXH56JKDU64UUR4LDODRHOI7GBKBLQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.927990350 (395h46m44.698631685s) [noise=true , address=77.90.31.19:4853 , piece_id=EBGKM4AN2HM376IQVGJI4HBXYV4SDU2DCOIEKHATCMNSJ44RSHEQ, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.928216142 (395h46m44.698857467s) [noise=true , address=77.90.31.16:4747 , piece_id=66IKFVQC5RZYEOSIH6KK4AMZPIP2POZOCESVLCYWCYWOJ3KHSWXA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:09.928439940 (395h46m44.699081275s) [noise=true , address=104.36.82.7:4927 , piece_id=CYPSRZUD7YT5ZV7FUXQOCNZW4Y7S7VIRELQT7IEL66NRQXESECCA, seg_pos={0 0}, upload=316661]: Uploading piece... Jul 9 20:57:11.491476539 (395h46m46.262117864s) [upload=316661]: Next returned nil segment Jul 9 20:57:11.491493962 (395h46m46.262135288s) [upload=316661]: Waiting for error group managing segments... Jul 9 20:57:11.913920504 (395h46m46.684561859s) [noise=true , address=77.90.31.21:4898 , piece_id=FVYXUCWSLIYZAREVWV2G2Z27Y2WDRUQK45OV7WZVGPHW5RS3GV7Q, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.914410241 (395h46m46.685051576s) [noise=true , address=77.90.31.11:4557 , piece_id=SG6TAPMMYIAJFWKK4HX77T3UNKP6DAGMMS72ZIMZL6LFV7CLYKGQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.914998206 (395h46m46.685639541s) [noise=true , address=79.127.219.38:4004 , piece_id=EN624ASKTAQ5RYXBLQZW5QAHMBBQPVTUSAJOGQVQKR4FX4GEBUIQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.929405148 (395h46m46.700046483s) [noise=true , address=77.90.31.14:4687 , piece_id=4LH6JKFAD26N3ZWUWPJCJHHO75FV7THBPGTLZYXV62ETHXNDQ7PQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.932087910 (395h46m46.702729245s) [noise=true , address=77.90.31.16:4747 , piece_id=66IKFVQC5RZYEOSIH6KK4AMZPIP2POZOCESVLCYWCYWOJ3KHSWXA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.937636836 (395h46m46.708278161s) [noise=true , address=77.90.31.19:4853 , piece_id=EBGKM4AN2HM376IQVGJI4HBXYV4SDU2DCOIEKHATCMNSJ44RSHEQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:11.939939832 (395h46m46.710581167s) [noise=true , address=77.90.31.19:4860 , piece_id=JA24EY5NBGW6IJUEALM3F45DQIXYXACFWAFELVOZCRNSHZXUJ5JQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:24.277638949 (395h46m59.048280284s) [noise=true , address=45.91.95.4:4012 , piece_id=IHTZ3AVW2K54AHPTCJTXSJDHKFQYRBJ6VMG3AWPMX4D6M7DSVWRQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:12kFYkJi2nr3EVpxZ6NVWcQwX8hRy3fXkfVmBNXjcmyPK2EoHMt, address:45.91.95.4:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:57:33.774824721 (395h47m8.545466046s) [noise=true , address=104.36.82.4:4832 , piece_id=YX42JF7UFDKDX2QXP4UM5KI62Q5BSN6IESQCV5OR3MZ5OOGMB4GQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:42.554846121 (395h47m17.325487456s) [noise=true , address=104.36.82.38:4708 , piece_id=LXSYAY3YTVT5Z2TWCGCQSZ372F5CP5L475DUI7LRCZLJFRADJJWQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:43.625027569 (395h47m18.395668904s) [noise=true , address=104.36.82.6:4880 , piece_id=PWT6SCZ4SGAQZD7NDQARHMVZNYDRHP5XV4U6E4LR5DMIUFWCF4CA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:51.567654956 (395h47m26.338296281s) [noise=true , address=104.36.82.40:4943 , piece_id=37XFBBMBJFKOEKIKSHSTA2CHMYK6I4Q54OOSX7IUL63R34IO5KFA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:57:52.866642294 (395h47m27.637283629s) [noise=true , address=104.36.82.7:4927 , piece_id=CYPSRZUD7YT5ZV7FUXQOCNZW4Y7S7VIRELQT7IEL66NRQXESECCA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: <nil> Jul 9 20:58:08.840679629 (395h47m43.611320964s) [noise=true , address=45.91.95.5:4002 , piece_id=EZ4RSL7IKDHQG2PLTJFR6N6YRQADFRNXVGUTLZYEWK5WJH3X65GA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:16MgCJPC1CuwnutbjFMaHYn7fmDzDoP7wyFYChw6m1WDKMLz4x, address:45.91.95.5:4002): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840712843 (395h47m43.611354168s) [noise=true , address=104.36.82.22:4513 , piece_id=CPDA5GKOYFVIBQCO3BEWZ7RXH56JKDU64UUR4LDODRHOI7GBKBLQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:104.36.82.22:4513): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840728242 (395h47m43.611369577s) [noise=true , address=104.36.82.51:4056 , piece_id=E3AJ35JU34YDAHZI432AOAYSCC7TMIND2V4HLHIJTQG5LCTJXWRQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1nY93g6WbZs2rfBYoKtf4Z7xGSr1QwizrBibKjpgKFbTbnVe1c, address:104.36.82.51:4056): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840744012 (395h47m43.611385347s) [noise=true , address=45.91.95.2:4005 , piece_id=YKKJEORN4UUZWLCOO3666DZ2FNGPASNHRUMTYDNF2MIJOSBTWLLA, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1bxFpBmz99gV4rL6bMtnHuthK2Xg4EusbfMAWF37RTRtPkqhQS, address:45.91.95.2:4005): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:58:08.840749212 (395h47m43.611390547s) [noise=true , address=45.91.95.3:4001 , piece_id=W7P3HQEG4RQEAPO2KAXHHMUMSQVZN62II56FLWCLJNORCUW27YKQ, seg_pos={0 0}, upload=316661]: Done uploading piece. err: ecclient: upload failed (node:1L6dYFtTZvGGF9XsUFDHBBaSNPTnYjUVPT7VU5hfVpJpmpn3jL, address:45.91.95.3:4001): protocol: write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection; write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51972->45.91.95.3:4001: use of closed network connection Jul 9 20:58:08.840772637 (395h47m43.611413972s) [seg_pos={0 0}, upload=316661]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 20:58:09.067780024 (395h47m43.838421389s) [upload=316661]: Done uploading segments... Jul 9 20:58:09.067830080 (395h47m43.838471415s) [upload=316661]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.228559543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..XX........X.X.........X........X........X.X...XXX.........] duration: 56.9291479s FAIL (2025-07-09 20:57:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.929172847s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X...................................X..X...] duration: 7.608611409s FAIL (2025-07-09 20:57:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.608635594s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X...................................X..X..] duration: 13.785403717s FAIL (2025-07-09 20:57:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.785425829s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X...................................X..X.] duration: 25.872317919s FAIL (2025-07-09 20:57:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.872339209s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [..........XX.........XX...X..X..X..X..X..X....X..X..........] duration: 56.942497694s FAIL (2025-07-09 20:57:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 505 (395h38m7.72866984s) [upload=316544]: Done starting upload Jul 9 20:48:32.958138646 (395h38m7.728779981s) [upload=316544]: Uploading segments... Jul 9 20:48:32.958260418 (395h38m7.728901753s) [upload=316544]: Got next segment. Inline: false Jul 9 20:48:32.958296448 (395h38m7.728937783s) [upload=316544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:32.958301678 (395h38m7.728943013s) [upload=316544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:33.408660883 (395h38m8.179302218s) [seg_pos={0 0}, upload=316544]: Begin upload segment... Jul 9 20:48:33.408714906 (395h38m8.179356251s) [seg_pos={0 0}, upload=316544]: Done begin upload segment. Jul 9 20:48:33.408750414 (395h38m8.179391749s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409126454 (395h38m8.179767789s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409390269 (395h38m8.180031604s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409650016 (395h38m8.180291341s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409901917 (395h38m8.180543252s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411382981 (395h38m8.182024326s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411571431 (395h38m8.182212787s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411880633 (395h38m8.182521968s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414121019 (395h38m8.184762374s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414150756 (395h38m8.184792091s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415293843 (395h38m8.185935188s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415710009 (395h38m8.186351344s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416024631 (395h38m8.186665966s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416425788 (395h38m8.187067123s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416744478 (395h38m8.187385823s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.417038510 (395h38m8.187679855s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418002505 (395h38m8.18864384s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418626729 (395h38m8.189268064s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:34.986954606 (395h38m9.757595931s) [upload=316544]: Next returned nil segment Jul 9 20:48:34.986977731 (395h38m9.757619066s) [upload=316544]: Waiting for error group managing segments... Jul 9 20:48:35.408257674 (395h38m10.17889901s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.408474730 (395h38m10.179116065s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.413258502 (395h38m10.183899837s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.421581956 (395h38m10.192223291s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422093875 (395h38m10.1927352s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422594053 (395h38m10.193235388s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.433967050 (395h38m10.204608385s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.435664839 (395h38m10.206306174s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.451281435 (395h38m10.22192277s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:37.814310008 (395h38m12.584951343s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:12MVRn8Rr1DV74CKDym2A8SFETLRWZntpjj2cHxgyQoRWeyqtD3, address:45.91.95.5:4020): protocol: storage node overloaded, request limit: 2000 Jul 9 20:48:47.755019185 (395h38m22.52566052s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:56.424997844 (395h38m31.195639179s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:06.796126718 (395h38m41.566768063s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:16.294257175 (395h38m51.0648985s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:32.310460328 (395h39m7.081101663s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310519180 (395h39m7.081160505s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp, address:104.36.82.22:4532): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310630493 (395h39m7.081271818s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP, address:45.91.95.2:4033): protocol: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection Jul 9 20:49:32.310665150 (395h39m7.081306485s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310696190 (395h39m7.081337525s) [seg_pos={0 0}, upload=316544]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:49:32.549829043 (395h39m7.320470368s) [upload=316544]: Done uploading segments... Jul 9 20:49:32.549855824 (395h39m7.320497149s) [upload=316544]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 56.942516579s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...XX........X.X.........X........X........X.X...XXX........] duration: 34.925708454s FAIL (2025-07-09 20:56:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.926042713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X...................................X..X] duration: 30.000849013s FAIL (2025-07-09 20:56:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000869933s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...........XX.........XX...X..X..X..X..X..X....X..X.........] duration: 36.684968642s FAIL (2025-07-09 20:56:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 505 (395h38m7.72866984s) [upload=316544]: Done starting upload Jul 9 20:48:32.958138646 (395h38m7.728779981s) [upload=316544]: Uploading segments... Jul 9 20:48:32.958260418 (395h38m7.728901753s) [upload=316544]: Got next segment. Inline: false Jul 9 20:48:32.958296448 (395h38m7.728937783s) [upload=316544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:32.958301678 (395h38m7.728943013s) [upload=316544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:33.408660883 (395h38m8.179302218s) [seg_pos={0 0}, upload=316544]: Begin upload segment... Jul 9 20:48:33.408714906 (395h38m8.179356251s) [seg_pos={0 0}, upload=316544]: Done begin upload segment. Jul 9 20:48:33.408750414 (395h38m8.179391749s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409126454 (395h38m8.179767789s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409390269 (395h38m8.180031604s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409650016 (395h38m8.180291341s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409901917 (395h38m8.180543252s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411382981 (395h38m8.182024326s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411571431 (395h38m8.182212787s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411880633 (395h38m8.182521968s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414121019 (395h38m8.184762374s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414150756 (395h38m8.184792091s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415293843 (395h38m8.185935188s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415710009 (395h38m8.186351344s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416024631 (395h38m8.186665966s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416425788 (395h38m8.187067123s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416744478 (395h38m8.187385823s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.417038510 (395h38m8.187679855s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418002505 (395h38m8.18864384s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418626729 (395h38m8.189268064s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:34.986954606 (395h38m9.757595931s) [upload=316544]: Next returned nil segment Jul 9 20:48:34.986977731 (395h38m9.757619066s) [upload=316544]: Waiting for error group managing segments... Jul 9 20:48:35.408257674 (395h38m10.17889901s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.408474730 (395h38m10.179116065s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.413258502 (395h38m10.183899837s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.421581956 (395h38m10.192223291s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422093875 (395h38m10.1927352s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422594053 (395h38m10.193235388s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.433967050 (395h38m10.204608385s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.435664839 (395h38m10.206306174s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.451281435 (395h38m10.22192277s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:37.814310008 (395h38m12.584951343s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:12MVRn8Rr1DV74CKDym2A8SFETLRWZntpjj2cHxgyQoRWeyqtD3, address:45.91.95.5:4020): protocol: storage node overloaded, request limit: 2000 Jul 9 20:48:47.755019185 (395h38m22.52566052s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:56.424997844 (395h38m31.195639179s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:06.796126718 (395h38m41.566768063s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:16.294257175 (395h38m51.0648985s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:32.310460328 (395h39m7.081101663s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310519180 (395h39m7.081160505s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp, address:104.36.82.22:4532): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310630493 (395h39m7.081271818s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP, address:45.91.95.2:4033): protocol: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection Jul 9 20:49:32.310665150 (395h39m7.081306485s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310696190 (395h39m7.081337525s) [seg_pos={0 0}, upload=316544]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:49:32.549829043 (395h39m7.320470368s) [upload=316544]: Done uploading segments... Jul 9 20:49:32.549855824 (395h39m7.320497149s) [upload=316544]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 36.684986797s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [....XX........X.X.........X........X........X.X...XXX.......] duration: 29.891486885s FAIL (2025-07-09 20:56:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.891508909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [............XX.........XX...X..X..X..X..X..X....X..X........] duration: 34.789194955s FAIL (2025-07-09 20:55:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 505 (395h38m7.72866984s) [upload=316544]: Done starting upload Jul 9 20:48:32.958138646 (395h38m7.728779981s) [upload=316544]: Uploading segments... Jul 9 20:48:32.958260418 (395h38m7.728901753s) [upload=316544]: Got next segment. Inline: false Jul 9 20:48:32.958296448 (395h38m7.728937783s) [upload=316544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:32.958301678 (395h38m7.728943013s) [upload=316544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:33.408660883 (395h38m8.179302218s) [seg_pos={0 0}, upload=316544]: Begin upload segment... Jul 9 20:48:33.408714906 (395h38m8.179356251s) [seg_pos={0 0}, upload=316544]: Done begin upload segment. Jul 9 20:48:33.408750414 (395h38m8.179391749s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409126454 (395h38m8.179767789s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409390269 (395h38m8.180031604s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409650016 (395h38m8.180291341s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409901917 (395h38m8.180543252s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411382981 (395h38m8.182024326s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411571431 (395h38m8.182212787s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411880633 (395h38m8.182521968s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414121019 (395h38m8.184762374s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414150756 (395h38m8.184792091s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415293843 (395h38m8.185935188s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415710009 (395h38m8.186351344s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416024631 (395h38m8.186665966s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416425788 (395h38m8.187067123s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416744478 (395h38m8.187385823s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.417038510 (395h38m8.187679855s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418002505 (395h38m8.18864384s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418626729 (395h38m8.189268064s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:34.986954606 (395h38m9.757595931s) [upload=316544]: Next returned nil segment Jul 9 20:48:34.986977731 (395h38m9.757619066s) [upload=316544]: Waiting for error group managing segments... Jul 9 20:48:35.408257674 (395h38m10.17889901s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.408474730 (395h38m10.179116065s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.413258502 (395h38m10.183899837s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.421581956 (395h38m10.192223291s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422093875 (395h38m10.1927352s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422594053 (395h38m10.193235388s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.433967050 (395h38m10.204608385s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.435664839 (395h38m10.206306174s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.451281435 (395h38m10.22192277s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:37.814310008 (395h38m12.584951343s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:12MVRn8Rr1DV74CKDym2A8SFETLRWZntpjj2cHxgyQoRWeyqtD3, address:45.91.95.5:4020): protocol: storage node overloaded, request limit: 2000 Jul 9 20:48:47.755019185 (395h38m22.52566052s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:56.424997844 (395h38m31.195639179s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:06.796126718 (395h38m41.566768063s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:16.294257175 (395h38m51.0648985s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:32.310460328 (395h39m7.081101663s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310519180 (395h39m7.081160505s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp, address:104.36.82.22:4532): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310630493 (395h39m7.081271818s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP, address:45.91.95.2:4033): protocol: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection Jul 9 20:49:32.310665150 (395h39m7.081306485s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310696190 (395h39m7.081337525s) [seg_pos={0 0}, upload=316544]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:49:32.549829043 (395h39m7.320470368s) [upload=316544]: Done uploading segments... Jul 9 20:49:32.549855824 (395h39m7.320497149s) [upload=316544]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.789220584s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.....XX........X.X.........X........X........X.X...XXX......] duration: 48.566212927s FAIL (2025-07-09 20:55:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.566228106s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...................................X..] duration: 6.217272258s FAIL (2025-07-09 20:54:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.217341451s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.............XX.........XX...X..X..X..X..X..X....X..X.......] duration: 43.940392368s FAIL (2025-07-09 20:54:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 505 (395h38m7.72866984s) [upload=316544]: Done starting upload Jul 9 20:48:32.958138646 (395h38m7.728779981s) [upload=316544]: Uploading segments... Jul 9 20:48:32.958260418 (395h38m7.728901753s) [upload=316544]: Got next segment. Inline: false Jul 9 20:48:32.958296448 (395h38m7.728937783s) [upload=316544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:32.958301678 (395h38m7.728943013s) [upload=316544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:33.408660883 (395h38m8.179302218s) [seg_pos={0 0}, upload=316544]: Begin upload segment... Jul 9 20:48:33.408714906 (395h38m8.179356251s) [seg_pos={0 0}, upload=316544]: Done begin upload segment. Jul 9 20:48:33.408750414 (395h38m8.179391749s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409126454 (395h38m8.179767789s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409390269 (395h38m8.180031604s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409650016 (395h38m8.180291341s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409901917 (395h38m8.180543252s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411382981 (395h38m8.182024326s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411571431 (395h38m8.182212787s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411880633 (395h38m8.182521968s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414121019 (395h38m8.184762374s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414150756 (395h38m8.184792091s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415293843 (395h38m8.185935188s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415710009 (395h38m8.186351344s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416024631 (395h38m8.186665966s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416425788 (395h38m8.187067123s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416744478 (395h38m8.187385823s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.417038510 (395h38m8.187679855s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418002505 (395h38m8.18864384s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418626729 (395h38m8.189268064s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:34.986954606 (395h38m9.757595931s) [upload=316544]: Next returned nil segment Jul 9 20:48:34.986977731 (395h38m9.757619066s) [upload=316544]: Waiting for error group managing segments... Jul 9 20:48:35.408257674 (395h38m10.17889901s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.408474730 (395h38m10.179116065s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.413258502 (395h38m10.183899837s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.421581956 (395h38m10.192223291s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422093875 (395h38m10.1927352s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422594053 (395h38m10.193235388s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.433967050 (395h38m10.204608385s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.435664839 (395h38m10.206306174s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.451281435 (395h38m10.22192277s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:37.814310008 (395h38m12.584951343s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:12MVRn8Rr1DV74CKDym2A8SFETLRWZntpjj2cHxgyQoRWeyqtD3, address:45.91.95.5:4020): protocol: storage node overloaded, request limit: 2000 Jul 9 20:48:47.755019185 (395h38m22.52566052s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:56.424997844 (395h38m31.195639179s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:06.796126718 (395h38m41.566768063s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:16.294257175 (395h38m51.0648985s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:32.310460328 (395h39m7.081101663s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310519180 (395h39m7.081160505s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp, address:104.36.82.22:4532): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310630493 (395h39m7.081271818s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP, address:45.91.95.2:4033): protocol: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection Jul 9 20:49:32.310665150 (395h39m7.081306485s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310696190 (395h39m7.081337525s) [seg_pos={0 0}, upload=316544]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:49:32.549829043 (395h39m7.320470368s) [upload=316544]: Done uploading segments... Jul 9 20:49:32.549855824 (395h39m7.320497149s) [upload=316544]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 43.940465257s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [......XX........X.X.........X........X........X.X...XXX.....] duration: 55.234331735s FAIL (2025-07-09 20:54:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 55.234342796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [..............XX.........XX...X..X..X..X..X..X....X..X......] duration: 38.479998709s FAIL (2025-07-09 20:53:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 505 (395h38m7.72866984s) [upload=316544]: Done starting upload Jul 9 20:48:32.958138646 (395h38m7.728779981s) [upload=316544]: Uploading segments... Jul 9 20:48:32.958260418 (395h38m7.728901753s) [upload=316544]: Got next segment. Inline: false Jul 9 20:48:32.958296448 (395h38m7.728937783s) [upload=316544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:32.958301678 (395h38m7.728943013s) [upload=316544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:33.408660883 (395h38m8.179302218s) [seg_pos={0 0}, upload=316544]: Begin upload segment... Jul 9 20:48:33.408714906 (395h38m8.179356251s) [seg_pos={0 0}, upload=316544]: Done begin upload segment. Jul 9 20:48:33.408750414 (395h38m8.179391749s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409126454 (395h38m8.179767789s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409390269 (395h38m8.180031604s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409650016 (395h38m8.180291341s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409901917 (395h38m8.180543252s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411382981 (395h38m8.182024326s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411571431 (395h38m8.182212787s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411880633 (395h38m8.182521968s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414121019 (395h38m8.184762374s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414150756 (395h38m8.184792091s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415293843 (395h38m8.185935188s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415710009 (395h38m8.186351344s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416024631 (395h38m8.186665966s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416425788 (395h38m8.187067123s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416744478 (395h38m8.187385823s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.417038510 (395h38m8.187679855s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418002505 (395h38m8.18864384s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418626729 (395h38m8.189268064s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:34.986954606 (395h38m9.757595931s) [upload=316544]: Next returned nil segment Jul 9 20:48:34.986977731 (395h38m9.757619066s) [upload=316544]: Waiting for error group managing segments... Jul 9 20:48:35.408257674 (395h38m10.17889901s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.408474730 (395h38m10.179116065s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.413258502 (395h38m10.183899837s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.421581956 (395h38m10.192223291s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422093875 (395h38m10.1927352s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422594053 (395h38m10.193235388s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.433967050 (395h38m10.204608385s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.435664839 (395h38m10.206306174s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.451281435 (395h38m10.22192277s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:37.814310008 (395h38m12.584951343s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:12MVRn8Rr1DV74CKDym2A8SFETLRWZntpjj2cHxgyQoRWeyqtD3, address:45.91.95.5:4020): protocol: storage node overloaded, request limit: 2000 Jul 9 20:48:47.755019185 (395h38m22.52566052s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:56.424997844 (395h38m31.195639179s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:06.796126718 (395h38m41.566768063s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:16.294257175 (395h38m51.0648985s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:32.310460328 (395h39m7.081101663s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310519180 (395h39m7.081160505s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp, address:104.36.82.22:4532): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310630493 (395h39m7.081271818s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP, address:45.91.95.2:4033): protocol: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection Jul 9 20:49:32.310665150 (395h39m7.081306485s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310696190 (395h39m7.081337525s) [seg_pos={0 0}, upload=316544]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:49:32.549829043 (395h39m7.320470368s) [upload=316544]: Done uploading segments... Jul 9 20:49:32.549855824 (395h39m7.320497149s) [upload=316544]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 38.480023968s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.......XX........X.X.........X........X........X.X...XXX....] duration: 33.228317574s FAIL (2025-07-09 20:53:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.228336018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...................................X.] duration: 6.863746467s FAIL (2025-07-09 20:53:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.863763601s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [........XX........X.X.........X........X........X.X...XXX...] duration: 43.465268885s FAIL (2025-07-09 20:53:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.465291639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...................................X] duration: 30.001081954s FAIL (2025-07-09 20:53:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001101792s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...............XX.........XX...X..X..X..X..X..X....X..X.....] duration: 30.245148836s FAIL (2025-07-09 20:52:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 505 (395h38m7.72866984s) [upload=316544]: Done starting upload Jul 9 20:48:32.958138646 (395h38m7.728779981s) [upload=316544]: Uploading segments... Jul 9 20:48:32.958260418 (395h38m7.728901753s) [upload=316544]: Got next segment. Inline: false Jul 9 20:48:32.958296448 (395h38m7.728937783s) [upload=316544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:32.958301678 (395h38m7.728943013s) [upload=316544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:33.408660883 (395h38m8.179302218s) [seg_pos={0 0}, upload=316544]: Begin upload segment... Jul 9 20:48:33.408714906 (395h38m8.179356251s) [seg_pos={0 0}, upload=316544]: Done begin upload segment. Jul 9 20:48:33.408750414 (395h38m8.179391749s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409126454 (395h38m8.179767789s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409390269 (395h38m8.180031604s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409650016 (395h38m8.180291341s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409901917 (395h38m8.180543252s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411382981 (395h38m8.182024326s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411571431 (395h38m8.182212787s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411880633 (395h38m8.182521968s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414121019 (395h38m8.184762374s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414150756 (395h38m8.184792091s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415293843 (395h38m8.185935188s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415710009 (395h38m8.186351344s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416024631 (395h38m8.186665966s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416425788 (395h38m8.187067123s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416744478 (395h38m8.187385823s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.417038510 (395h38m8.187679855s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418002505 (395h38m8.18864384s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418626729 (395h38m8.189268064s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:34.986954606 (395h38m9.757595931s) [upload=316544]: Next returned nil segment Jul 9 20:48:34.986977731 (395h38m9.757619066s) [upload=316544]: Waiting for error group managing segments... Jul 9 20:48:35.408257674 (395h38m10.17889901s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.408474730 (395h38m10.179116065s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.413258502 (395h38m10.183899837s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.421581956 (395h38m10.192223291s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422093875 (395h38m10.1927352s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422594053 (395h38m10.193235388s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.433967050 (395h38m10.204608385s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.435664839 (395h38m10.206306174s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.451281435 (395h38m10.22192277s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:37.814310008 (395h38m12.584951343s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:12MVRn8Rr1DV74CKDym2A8SFETLRWZntpjj2cHxgyQoRWeyqtD3, address:45.91.95.5:4020): protocol: storage node overloaded, request limit: 2000 Jul 9 20:48:47.755019185 (395h38m22.52566052s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:56.424997844 (395h38m31.195639179s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:06.796126718 (395h38m41.566768063s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:16.294257175 (395h38m51.0648985s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:32.310460328 (395h39m7.081101663s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310519180 (395h39m7.081160505s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp, address:104.36.82.22:4532): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310630493 (395h39m7.081271818s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP, address:45.91.95.2:4033): protocol: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection Jul 9 20:49:32.310665150 (395h39m7.081306485s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310696190 (395h39m7.081337525s) [seg_pos={0 0}, upload=316544]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:49:32.549829043 (395h39m7.320470368s) [upload=316544]: Done uploading segments... Jul 9 20:49:32.549855824 (395h39m7.320497149s) [upload=316544]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 30.245166611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X........XX........X.X.........X........X........X.X...XXX..] duration: 38.406884965s FAIL (2025-07-09 20:52:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.406905514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [................XX.........XX...X..X..X..X..X..X....X..X....] duration: 49.617750494s FAIL (2025-07-09 20:52:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 505 (395h38m7.72866984s) [upload=316544]: Done starting upload Jul 9 20:48:32.958138646 (395h38m7.728779981s) [upload=316544]: Uploading segments... Jul 9 20:48:32.958260418 (395h38m7.728901753s) [upload=316544]: Got next segment. Inline: false Jul 9 20:48:32.958296448 (395h38m7.728937783s) [upload=316544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:32.958301678 (395h38m7.728943013s) [upload=316544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:33.408660883 (395h38m8.179302218s) [seg_pos={0 0}, upload=316544]: Begin upload segment... Jul 9 20:48:33.408714906 (395h38m8.179356251s) [seg_pos={0 0}, upload=316544]: Done begin upload segment. Jul 9 20:48:33.408750414 (395h38m8.179391749s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409126454 (395h38m8.179767789s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409390269 (395h38m8.180031604s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409650016 (395h38m8.180291341s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409901917 (395h38m8.180543252s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411382981 (395h38m8.182024326s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411571431 (395h38m8.182212787s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411880633 (395h38m8.182521968s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414121019 (395h38m8.184762374s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414150756 (395h38m8.184792091s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415293843 (395h38m8.185935188s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415710009 (395h38m8.186351344s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416024631 (395h38m8.186665966s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416425788 (395h38m8.187067123s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416744478 (395h38m8.187385823s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.417038510 (395h38m8.187679855s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418002505 (395h38m8.18864384s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418626729 (395h38m8.189268064s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:34.986954606 (395h38m9.757595931s) [upload=316544]: Next returned nil segment Jul 9 20:48:34.986977731 (395h38m9.757619066s) [upload=316544]: Waiting for error group managing segments... Jul 9 20:48:35.408257674 (395h38m10.17889901s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.408474730 (395h38m10.179116065s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.413258502 (395h38m10.183899837s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.421581956 (395h38m10.192223291s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422093875 (395h38m10.1927352s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422594053 (395h38m10.193235388s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.433967050 (395h38m10.204608385s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.435664839 (395h38m10.206306174s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.451281435 (395h38m10.22192277s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:37.814310008 (395h38m12.584951343s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:12MVRn8Rr1DV74CKDym2A8SFETLRWZntpjj2cHxgyQoRWeyqtD3, address:45.91.95.5:4020): protocol: storage node overloaded, request limit: 2000 Jul 9 20:48:47.755019185 (395h38m22.52566052s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:56.424997844 (395h38m31.195639179s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:06.796126718 (395h38m41.566768063s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:16.294257175 (395h38m51.0648985s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:32.310460328 (395h39m7.081101663s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310519180 (395h39m7.081160505s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp, address:104.36.82.22:4532): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310630493 (395h39m7.081271818s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP, address:45.91.95.2:4033): protocol: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection Jul 9 20:49:32.310665150 (395h39m7.081306485s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310696190 (395h39m7.081337525s) [seg_pos={0 0}, upload=316544]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:49:32.549829043 (395h39m7.320470368s) [upload=316544]: Done uploading segments... Jul 9 20:49:32.549855824 (395h39m7.320497149s) [upload=316544]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 49.617765953s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.939590247s FAIL (2025-07-09 20:51:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.939613943s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.................XX.........XX...X..X..X..X..X..X....X..X...] duration: 41.020676674s FAIL (2025-07-09 20:51:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 505 (395h38m7.72866984s) [upload=316544]: Done starting upload Jul 9 20:48:32.958138646 (395h38m7.728779981s) [upload=316544]: Uploading segments... Jul 9 20:48:32.958260418 (395h38m7.728901753s) [upload=316544]: Got next segment. Inline: false Jul 9 20:48:32.958296448 (395h38m7.728937783s) [upload=316544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:48:32.958301678 (395h38m7.728943013s) [upload=316544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:48:33.408660883 (395h38m8.179302218s) [seg_pos={0 0}, upload=316544]: Begin upload segment... Jul 9 20:48:33.408714906 (395h38m8.179356251s) [seg_pos={0 0}, upload=316544]: Done begin upload segment. Jul 9 20:48:33.408750414 (395h38m8.179391749s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409126454 (395h38m8.179767789s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409390269 (395h38m8.180031604s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409650016 (395h38m8.180291341s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.409901917 (395h38m8.180543252s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411382981 (395h38m8.182024326s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411571431 (395h38m8.182212787s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.411880633 (395h38m8.182521968s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414121019 (395h38m8.184762374s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.414150756 (395h38m8.184792091s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415293843 (395h38m8.185935188s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.415710009 (395h38m8.186351344s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416024631 (395h38m8.186665966s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416425788 (395h38m8.187067123s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.416744478 (395h38m8.187385823s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.417038510 (395h38m8.187679855s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418002505 (395h38m8.18864384s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:33.418626729 (395h38m8.189268064s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Uploading piece... Jul 9 20:48:34.986954606 (395h38m9.757595931s) [upload=316544]: Next returned nil segment Jul 9 20:48:34.986977731 (395h38m9.757619066s) [upload=316544]: Waiting for error group managing segments... Jul 9 20:48:35.408257674 (395h38m10.17889901s) [noise=true , address=77.90.31.18:4829 , piece_id=OHSQSEBQQKUM6QTR7YOZEPWYAUJ2U3LYRMBL33JL4DNKT6DMVOXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.408474730 (395h38m10.179116065s) [noise=true , address=77.90.31.14:4669 , piece_id=WDBRS35XZMGBM62QJWU36CVZZ6QGZXL5ETJMLFQSVBLF4FNHZAOQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.413258502 (395h38m10.183899837s) [noise=true , address=77.90.31.17:4784 , piece_id=X54YUFDI5QH5CONPBNZ6AALSNWY6MOFZPRGGDOJRZB5XBRPYMM7Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.421581956 (395h38m10.192223291s) [noise=true , address=77.90.31.21:4883 , piece_id=B446EWKA7ZWU6F62YG6RRE2QS657HQW2XFZSDADS6CNBIMP6A4WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422093875 (395h38m10.1927352s) [noise=true , address=77.90.31.15:4722 , piece_id=6LGIWYYRJPFSGTWESKKUEEUYRB4ZEITBWVD3SW7HRJ2L4ZQNVWPQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.422594053 (395h38m10.193235388s) [noise=true , address=77.90.31.18:4815 , piece_id=H4TXRY5UPATMM5NXVE35N54XKZMW4UISTBUMFQR7ZJR4QITWYYXQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.433967050 (395h38m10.204608385s) [noise=true , address=77.90.31.13:4645 , piece_id=QYMEELRBJHSRL5CXLDL57NN25NODR72LCNVLIZSX226U6FSMITXA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.435664839 (395h38m10.206306174s) [noise=true , address=77.90.31.19:4853 , piece_id=KJLZZNCJZHWOVS76VN4S5C7NX4XXUAUZTNN2UODY7EM3MJKWNT5Q, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:35.451281435 (395h38m10.22192277s) [noise=true , address=79.127.219.38:4002 , piece_id=5MHKJ34D5JQ4VPXOZEG2JCOX7VFKWR35ZR7AUNYIR747UXJIE4PA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:37.814310008 (395h38m12.584951343s) [noise=true , address=45.91.95.5:4020 , piece_id=J6INMBHE6ZQWFRSIEB2WQFWQQVJKZYM5GIEABBRWQ24NWUHILBWA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:12MVRn8Rr1DV74CKDym2A8SFETLRWZntpjj2cHxgyQoRWeyqtD3, address:45.91.95.5:4020): protocol: storage node overloaded, request limit: 2000 Jul 9 20:48:47.755019185 (395h38m22.52566052s) [noise=true , address=104.36.82.51:4064 , piece_id=F47YKLET4X3PNRWE46NCGMN57QDLQDOF25EXUNLLAULJEJ23VNUA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:48:56.424997844 (395h38m31.195639179s) [noise=true , address=45.91.95.4:4034 , piece_id=3VQOACVJNOUCLDG5HBSBMYFVOXCVN5B4LCDXKSE55ZKPW3QIPUVQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:06.796126718 (395h38m41.566768063s) [noise=true , address=104.36.82.6:4884 , piece_id=BFNMLWX2KYBZTRVQTGSLCJE2RZ5677BIEC56RMMLIH2B56AIQIQA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:16.294257175 (395h38m51.0648985s) [noise=true , address=104.36.82.40:4949 , piece_id=UL7VPNW72NOEFCKXUOCDPOD6ROA4GOZXLAQOYOGGX3DMHSK4D2XA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: <nil> Jul 9 20:49:32.310460328 (395h39m7.081101663s) [noise=true , address=45.91.95.3:4015 , piece_id=6OUNQGFFWKKUXXL24RFXY5QXCN3ESAY4L3YHT6UARAZO7ZYL72WA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U, address:45.91.95.3:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310519180 (395h39m7.081160505s) [noise=true , address=104.36.82.22:4532 , piece_id=CQLJ2XNEEA33NEXXBUFX7F4ZQPHYMLRQPAMGUQEMILDWUIFHDROQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp, address:104.36.82.22:4532): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310630493 (395h39m7.081271818s) [noise=true , address=45.91.95.2:4033 , piece_id=QZQX54XTYL34FYNBPGXU2PVHDN7WELL6IDDEBNXYSLPLQXCN5AYQ, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1bQEzLer6oNTKcy7KCuDqmm8CiJoTCZwSQJqiEeFrADcLGyGxP, address:45.91.95.2:4033): protocol: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41756->45.91.95.2:4033: use of closed network connection Jul 9 20:49:32.310665150 (395h39m7.081306485s) [noise=true , address=104.36.82.36:4628 , piece_id=Z2CCU7XVYMBODMECEMXUJTX55Y3KUAFAC7KTKAVUWM5GLZNGKQLA, seg_pos={0 0}, upload=316544]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:49:32.310696190 (395h39m7.081337525s) [seg_pos={0 0}, upload=316544]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:49:32.549829043 (395h39m7.320470368s) [upload=316544]: Done uploading segments... Jul 9 20:49:32.549855824 (395h39m7.320497149s) [upload=316544]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 41.020704659s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.X........XX........X.X.........X........X........X.X...XXX.] duration: 45.445411064s FAIL (2025-07-09 20:51:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.445424339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.177599782s FAIL (2025-07-09 20:50:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.177621273s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [..X........XX........X.X.........X........X........X.X...XXX] duration: 1m0.001318243s FAIL (2025-07-09 20:50:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001333443s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)