Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.864016957s FAIL (2025-07-26 23:33:28 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.86404975s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.239046085s FAIL (2025-07-26 23:33:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.239073707s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [X...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X.] duration: 50.842401933s FAIL (2025-07-26 23:33:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.84244684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.237723232s FAIL (2025-07-26 23:32:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== : Uploading piece... Jul 26 23:31:57.342681704 (59h56m7.065244744s) [noise=true , address=79.127.205.235:4689 , piece_id=JISSTIHBZ244SY2PLPCAARM4QWIGY4OCKFDD44ZIAWZZGGSP5DXQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.343141715 (59h56m7.065704765s) [noise=true , address=77.90.31.21:4902 , piece_id=M2XNP67Q6JSMGHTBDGXUEDWGXOFDA2SRANN3ATVKHJVMZESOMBRQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.346340565 (59h56m7.068903605s) [noise=true , address=77.90.31.12:4601 , piece_id=RTSRAMSSUKHR6QCAB4HCNTH4QGLM3HR4PAAA6LDI2WCOLDVADQGA, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.346494079 (59h56m7.069057119s) [noise=true , address=104.36.82.46:4494 , piece_id=5H4RKG6LLFKG6YCW74MGOICDXJFBUAX2ZWVXNW6TT6EE3JNXDA4Q, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.346672069 (59h56m7.069235109s) [noise=true , address=79.127.205.246:4270 , piece_id=XQVNALQ4O24R47VUYFZHXAPTWGLZJP5NGUMVYHVSHCS2ZVT6P3SQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.346870759 (59h56m7.069433799s) [noise=true , address=104.36.82.56:4241 , piece_id=AOI76JZBN42YFC76HWUGCGMMB2ZCSMFVJMEND2ISMWOKQXL6DF7Q, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.346968627 (59h56m7.069531667s) [noise=true , address=156.146.43.225:4448 , piece_id=34Q6V4DXM6NSS62WEGG4OXT7CRYFXWRUVSVP4NJXUPN4NW2NCXGA, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.347192304 (59h56m7.069755344s) [noise=true , address=77.90.31.17:4793 , piece_id=I7FX3KTIYNIIDBKVF772FCU44HGUKOI64PZDOKZKXFTU7BYKG37A, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.347265986 (59h56m7.069829036s) [noise=true , address=79.127.220.98:4422 , piece_id=YHVHZMXXBBBMH2YDVPJHMYJKXTTVQLLZ4RUQYUOWHKZIK5AFPADQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.347559707 (59h56m7.070122747s) [noise=true , address=104.36.82.47:4549 , piece_id=Z7THAJAOWRQQYIOBAKN3WQQUFZSGW7X6FRCD3IIBQPLJIDQYGW4A, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.347600175 (59h56m7.070163215s) [noise=true , address=79.127.201.215:4353 , piece_id=J3C5WCMWXBBEKL3IJKDZXT4VQQPZWBJS7NDUGSTRVNIHTB5DVXZQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.347848069 (59h56m7.070411109s) [noise=true , address=104.36.82.41:4791 , piece_id=VS3QTUH4XEENVEGAJM56BMIE4MIQMY3QJS4R3PXSFMMIB3SGJJKQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.348135328 (59h56m7.070698368s) [noise=true , address=45.91.116.9:4277 , piece_id=5427THO2OLAB5EESBTNTCEAO5GOEQS2EX7HDTUDME5LBZBGHJCDQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.348479177 (59h56m7.071042217s) [noise=true , address=64.49.13.140:4508 , piece_id=MODQHDBOKHHJHRJPLOIEQQGIAGIMRYSFMXVWUGXLRYW2ME72AARQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.348791374 (59h56m7.071354414s) [noise=true , address=79.127.219.51:4212 , piece_id=6G32LZJ3I63G3V5E6HIZBXSE35XRW57I5ZQMW72QAK4COCIJDPHQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.349085086 (59h56m7.071648126s) [noise=true , address=79.127.205.245:4239 , piece_id=4IL3DJMHGT7SFMRDRRQIJQ6MHMOYROKJFSSKEXOFQDLRDQKKPASA, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.349275811 (59h56m7.071838851s) [noise=true , address=104.36.82.54:4180 , piece_id=PFRJU4N3MMXIZYQWTPA2F75CN3HX5SMZDJVIUMCQDY7GMIGBHFTQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:58.453333466 (59h56m8.175896496s) [noise=true , address=156.146.43.225:4448 , piece_id=34Q6V4DXM6NSS62WEGG4OXT7CRYFXWRUVSVP4NJXUPN4NW2NCXGA, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 23:31:58.457343028 (59h56m8.179906068s) [upload=57500]: Next returned nil segment Jul 26 23:31:58.457354711 (59h56m8.179917751s) [upload=57500]: Waiting for error group managing segments... Jul 26 23:31:58.759161939 (59h56m8.48172498s) [noise=true , address=79.127.205.245:4239 , piece_id=4IL3DJMHGT7SFMRDRRQIJQ6MHMOYROKJFSSKEXOFQDLRDQKKPASA, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:31:58.771094599 (59h56m8.493657639s) [noise=true , address=79.127.205.235:4689 , piece_id=JISSTIHBZ244SY2PLPCAARM4QWIGY4OCKFDD44ZIAWZZGGSP5DXQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:31:58.800868877 (59h56m8.523431917s) [noise=true , address=79.127.205.246:4270 , piece_id=XQVNALQ4O24R47VUYFZHXAPTWGLZJP5NGUMVYHVSHCS2ZVT6P3SQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:31:59.130345965 (59h56m8.852909005s) [noise=true , address=79.127.220.98:4422 , piece_id=YHVHZMXXBBBMH2YDVPJHMYJKXTTVQLLZ4RUQYUOWHKZIK5AFPADQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:31:59.350051629 (59h56m9.072614669s) [noise=true , address=77.90.31.21:4902 , piece_id=M2XNP67Q6JSMGHTBDGXUEDWGXOFDA2SRANN3ATVKHJVMZESOMBRQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:32:00.663326730 (59h56m10.38588977s) [noise=true , address=79.127.219.51:4212 , piece_id=6G32LZJ3I63G3V5E6HIZBXSE35XRW57I5ZQMW72QAK4COCIJDPHQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:32:01.250947466 (59h56m10.973510506s) [noise=true , address=64.49.13.140:4508 , piece_id=MODQHDBOKHHJHRJPLOIEQQGIAGIMRYSFMXVWUGXLRYW2ME72AARQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:32:01.842499864 (59h56m11.565062914s) [noise=true , address=79.127.220.101:4048 , piece_id=VN65WZ3Z24O5RIIXXH4ATRHWJ2JFRGSICNWZGD6R7ZNYCJL4N5LA, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:32:02.379906966 (59h56m12.102470006s) [noise=true , address=45.91.116.9:4277 , piece_id=5427THO2OLAB5EESBTNTCEAO5GOEQS2EX7HDTUDME5LBZBGHJCDQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:32:26.935346561 (59h56m36.657909651s) [noise=true , address=79.127.201.215:4353 , piece_id=J3C5WCMWXBBEKL3IJKDZXT4VQQPZWBJS7NDUGSTRVNIHTB5DVXZQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:32:56.238172780 (59h57m5.96073582s) [noise=true , address=104.36.82.46:4494 , piece_id=5H4RKG6LLFKG6YCW74MGOICDXJFBUAX2ZWVXNW6TT6EE3JNXDA4Q, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: rpc: dial tcp 104.36.82.46:4494: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238215362 (59h57m5.960778402s) [noise=true , address=77.90.31.12:4601 , piece_id=RTSRAMSSUKHR6QCAB4HCNTH4QGLM3HR4PAAA6LDI2WCOLDVADQGA, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2, address:77.90.31.12:4601): protocol: rpc: dial tcp 77.90.31.12:4601: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238233165 (59h57m5.960796205s) [noise=true , address=104.36.82.54:4180 , piece_id=PFRJU4N3MMXIZYQWTPA2F75CN3HX5SMZDJVIUMCQDY7GMIGBHFTQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: rpc: dial tcp 104.36.82.54:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238252192 (59h57m5.960815232s) [noise=true , address=104.36.82.56:4241 , piece_id=AOI76JZBN42YFC76HWUGCGMMB2ZCSMFVJMEND2ISMWOKQXL6DF7Q, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:12tDLUhSXGgBTAJ5n3LWRtP53pJfmkUnDu7CG1kmMXbFpSkHwQ8, address:104.36.82.56:4241): protocol: write tcp 188.166.225.170:47248->104.36.82.56:4241: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238274856 (59h57m5.960838066s) [noise=true , address=104.36.82.47:4549 , piece_id=Z7THAJAOWRQQYIOBAKN3WQQUFZSGW7X6FRCD3IIBQPLJIDQYGW4A, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:1HWoyMKQ6v2XWUfEmfFh3m8AiGivAcYkkZ9na8kqHFzH2hPDeK, address:104.36.82.47:4549): protocol: rpc: dial tcp 104.36.82.47:4549: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238326414 (59h57m5.960889454s) [noise=true , address=104.36.82.41:4791 , piece_id=VS3QTUH4XEENVEGAJM56BMIE4MIQMY3QJS4R3PXSFMMIB3SGJJKQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:12LpFW1mfotWPqgQxyBSimChkttujx21L6rU2GRSwvqbZ7CVeLn, address:104.36.82.41:4791): protocol: write tcp 188.166.225.170:39438->104.36.82.41:4791: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238347835 (59h57m5.960910876s) [noise=true , address=77.90.31.17:4793 , piece_id=I7FX3KTIYNIIDBKVF772FCU44HGUKOI64PZDOKZKXFTU7BYKG37A, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: read tcp 188.166.225.170:40560->77.90.31.17:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:32:56.238368554 (59h57m5.960931594s) [seg_pos={0 0}, upload=57500]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:32:56.474209089 (59h57m6.196772129s) [upload=57500]: Done uploading segments... Jul 26 23:32:56.474239567 (59h57m6.196802597s) [upload=57500]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.237743762s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...........................XXX.X..X.....XXXXX...XXX.X.....X.] duration: 8.86876745s FAIL (2025-07-26 23:32:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.868789893s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.....X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235877175s FAIL (2025-07-26 23:32:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== : dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205331346 (59h56m31.927894386s) [seg_pos={0 0}, upload=57499]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:32:22.440397036 (59h56m32.162960066s) [upload=57499]: Done uploading segments... Jul 26 23:32:22.440419980 (59h56m32.16298301s) [upload=57499]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:32:23.313259465 (59h56m33.035822505s) [upload=57508]: Starting upload Jul 26 23:32:23.313324229 (59h56m33.035887259s) [upload=57508]: Done starting upload Jul 26 23:32:23.313412177 (59h56m33.035975257s) [upload=57508]: Uploading segments... Jul 26 23:32:23.313574407 (59h56m33.036137447s) [upload=57508]: Got next segment. Inline: false Jul 26 23:32:23.313604294 (59h56m33.036167334s) [upload=57508]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:32:23.313610226 (59h56m33.036173266s) [upload=57508]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:32:23.761321834 (59h56m33.483884864s) [seg_pos={0 0}, upload=57508]: Begin upload segment... Jul 26 23:32:23.761361420 (59h56m33.48392446s) [seg_pos={0 0}, upload=57508]: Done begin upload segment. Jul 26 23:32:23.761377391 (59h56m33.483940431s) [noise=true , address=77.90.31.12:4592 , piece_id=IQY6RS73BDSRIHH4O4JKA4BTHM2S2BQAQ5ZPOTQ4FHV2TYMR6OKQ, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.761388552 (59h56m33.483951592s) [noise=true , address=104.36.82.11:4113 , piece_id=4IBHWBVZKUEQ3XH35W67Z2UXPSJCNDTAQT5F4EEUXM57EHW56XQQ, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.761729774 (59h56m33.484292894s) [noise=true , address=45.91.116.4:4939 , piece_id=UUCMHDCUJQGX2ERLROFNPWGADS3XZLTHMSX65MIZC7ZLNM66EHOA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762254168 (59h56m33.484817208s) [noise=true , address=45.91.116.6:4973 , piece_id=Y3NW4UNL3LCPYN4A5ZISMU3AWHYVUEY7H74PESGUCPC7ERLOBX7Q, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762382954 (59h56m33.484945994s) [noise=true , address=104.36.82.6:4875 , piece_id=FNQMYIEKWGG525N7TXWHE7UV7DGOAFAK32HRCHH4D4RZVWH7NH6Q, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762526849 (59h56m33.485089889s) [noise=true , address=77.90.31.19:4843 , piece_id=4PIBA2ORYW73FITBT3WVS373REJTRQGAWPJ3RNZB3C3QI3TI6DAA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762627141 (59h56m33.485190191s) [noise=true , address=77.90.31.21:4877 , piece_id=SXPA3ETVDY6RFUASZLBY4C2URCEQ33SQXGOT2G3KDDDSX6UOMUDA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762797687 (59h56m33.485360727s) [noise=true , address=45.91.116.7:4010 , piece_id=XYFI47RUCFZLPVOVISSYY7MTUOPMGNI2BDK5JE2S25OJWLOSOD2A, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762842854 (59h56m33.485405904s) [noise=true , address=104.36.82.5:4837 , piece_id=V7JD52W3A5JMKDNOESLPMNIHEW4OZO64KUHZABDSYPVWNNI65AJA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.763061172 (59h56m33.485624212s) [noise=true , address=104.36.82.15:4259 , piece_id=V34UN7R3KBZJC6GCOPUKXBCKDPBNTFWRDRVJCCBLNF5MQ3KPHJPQ, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.763322842 (59h56m33.485885882s) [noise=true , address=45.91.116.14:4189 , piece_id=M26C25Q5KZECJMSQMVY5KHXU4MKVTGDZQ7LXUGIPOCIKV2EROLFA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.763562401 (59h56m33.486125441s) [noise=true , address=45.91.116.13:4154 , piece_id=DHQRRAX77RWMMQDUVODWPTEI7KTXOPTMWZ4QOGF776PS24EN3HUA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.763800727 (59h56m33.486363757s) [noise=true , address=104.36.82.2:4731 , piece_id=VPU4KGAJVVOJVXXJ4KSYWPA4KMRQAPSHZR3GUA5NT7MNAFA5L7UQ, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.763896590 (59h56m33.48645962s) [noise=true , address=45.91.116.5:4299 , piece_id=TYMR2N4K35PSGU7GV2YL5JTVOU47X2UVFTPZBKBO3DJYVVPVTJ6A, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.764115619 (59h56m33.486678659s) [noise=true , address=45.91.116.7:4011 , piece_id=BA4YCAJPOHVJBQOIK6IPN7K532F7M6O7WO5CRDX5KX6MYUOSXA5Q, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.764350058 (59h56m33.486913098s) [noise=true , address=77.90.31.16:4733 , piece_id=5VRUIEUVCPT2EDPWOXMN45CJAPBJXUD5C4NPJOTTVGDIKUEKSEKQ, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:25.490129184 (59h56m35.212692224s) [upload=57508]: Next returned nil segment Jul 26 23:32:25.490145345 (59h56m35.212708385s) [upload=57508]: Waiting for error group managing segments... Jul 26 23:32:25.775412808 (59h56m35.497975848s) [noise=true , address=45.91.116.7:4011 , piece_id=BA4YCAJPOHVJBQOIK6IPN7K532F7M6O7WO5CRDX5KX6MYUOSXA5Q, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.851963552 (59h56m35.574526592s) [noise=true , address=45.91.116.7:4010 , piece_id=XYFI47RUCFZLPVOVISSYY7MTUOPMGNI2BDK5JE2S25OJWLOSOD2A, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.862929614 (59h56m35.585492654s) [noise=true , address=45.91.116.13:4154 , piece_id=DHQRRAX77RWMMQDUVODWPTEI7KTXOPTMWZ4QOGF776PS24EN3HUA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.864755447 (59h56m35.587318487s) [noise=true , address=45.91.116.4:4939 , piece_id=UUCMHDCUJQGX2ERLROFNPWGADS3XZLTHMSX65MIZC7ZLNM66EHOA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.906462595 (59h56m35.629025625s) [noise=true , address=45.91.116.14:4189 , piece_id=M26C25Q5KZECJMSQMVY5KHXU4MKVTGDZQ7LXUGIPOCIKV2EROLFA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.929984891 (59h56m35.652547931s) [noise=true , address=45.91.116.6:4973 , piece_id=Y3NW4UNL3LCPYN4A5ZISMU3AWHYVUEY7H74PESGUCPC7ERLOBX7Q, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.950169380 (59h56m35.67273242s) [noise=true , address=77.90.31.16:4733 , piece_id=5VRUIEUVCPT2EDPWOXMN45CJAPBJXUD5C4NPJOTTVGDIKUEKSEKQ, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.951688737 (59h56m35.674251767s) [noise=true , address=45.91.116.5:4299 , piece_id=TYMR2N4K35PSGU7GV2YL5JTVOU47X2UVFTPZBKBO3DJYVVPVTJ6A, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:26.078913226 (59h56m35.801476256s) [noise=true , address=77.90.31.21:4877 , piece_id=SXPA3ETVDY6RFUASZLBY4C2URCEQ33SQXGOT2G3KDDDSX6UOMUDA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:27.366812726 (59h56m37.089375756s) [noise=true , address=77.90.31.19:4843 , piece_id=4PIBA2ORYW73FITBT3WVS373REJTRQGAWPJ3RNZB3C3QI3TI6DAA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:52.666109045 (59h57m2.388672085s) [noise=true , address=104.36.82.5:4837 , piece_id=V7JD52W3A5JMKDNOESLPMNIHEW4OZO64KUHZABDSYPVWNNI65AJA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666195561 (59h57m2.388758641s) [noise=true , address=104.36.82.6:4875 , piece_id=FNQMYIEKWGG525N7TXWHE7UV7DGOAFAK32HRCHH4D4RZVWH7NH6Q, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666246458 (59h57m2.388809498s) [noise=true , address=104.36.82.15:4259 , piece_id=V34UN7R3KBZJC6GCOPUKXBCKDPBNTFWRDRVJCCBLNF5MQ3KPHJPQ, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666284491 (59h57m2.388847531s) [noise=true , address=77.90.31.12:4592 , piece_id=IQY6RS73BDSRIHH4O4JKA4BTHM2S2BQAQ5ZPOTQ4FHV2TYMR6OKQ, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666293388 (59h57m2.388856428s) [noise=true , address=104.36.82.2:4731 , piece_id=VPU4KGAJVVOJVXXJ4KSYWPA4KMRQAPSHZR3GUA5NT7MNAFA5L7UQ, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666300751 (59h57m2.388863792s) [noise=true , address=104.36.82.11:4113 , piece_id=4IBHWBVZKUEQ3XH35W67Z2UXPSJCNDTAQT5F4EEUXM57EHW56XQQ, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:47632->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666325309 (59h57m2.388888339s) [seg_pos={0 0}, upload=57508]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:32:52.900500787 (59h57m2.623063827s) [upload=57508]: Done uploading segments... Jul 26 23:32:52.900544341 (59h57m2.623107371s) [upload=57508]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.235897484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [X...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.234367996s FAIL (2025-07-26 23:32:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.234388957s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.868340249s FAIL (2025-07-26 23:32:38 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.868367541s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X] duration: 1m0.001463301s FAIL (2025-07-26 23:32:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001474952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [......X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.236924701s FAIL (2025-07-26 23:32:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== tch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:31:52.845052411 (59h56m2.567615441s) [upload=57499]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:31:53.079527764 (59h56m2.802090804s) [seg_pos={0 0}, upload=57499]: Begin upload segment... Jul 26 23:31:53.079567780 (59h56m2.80213082s) [seg_pos={0 0}, upload=57499]: Done begin upload segment. Jul 26 23:31:53.079591957 (59h56m2.802154987s) [noise=true , address=45.91.116.12:4225 , piece_id=TEHDDMAKC7P2GHBJI4E34W27YZ3CRLZSZCD7B2A7FQW5MUS5SVFA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.079639598 (59h56m2.802202638s) [noise=true , address=77.90.31.17:4770 , piece_id=HJFXXZV5TX2FZZF4LFUJT526D66IYHQTEJ6YTUNVRIWV4D3DACPA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080010728 (59h56m2.802573768s) [noise=true , address=45.91.116.8:4045 , piece_id=QXULBJBY73CKXBNNRRSQWSQKRU4L2MKLGTJVTJEF3N7LMH47AQTQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080147349 (59h56m2.802710389s) [noise=true , address=104.36.82.3:4765 , piece_id=OLEWJ6R5VRNNRWVQUYX3KSG5GANPRBXW23M55OO6KSW7XQD74BRA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080305833 (59h56m2.802868873s) [noise=true , address=104.36.82.12:4150 , piece_id=YIH7HSCLB7V5OARSUKSMTREKTEBTS6S4UYEUBE7PQXBLNIW77W2A, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080573715 (59h56m2.803136755s) [noise=true , address=45.91.116.10:4082 , piece_id=JLVQYPCY6FEJJ54YPX5GHMMQTDYEOT6PIBAHL4DWXU2WH3TJ25VQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080628801 (59h56m2.803191831s) [noise=true , address=104.36.82.7:4910 , piece_id=X6I37G3J7KPYZVBOXGEADLTECWJRURRVVW5TKGRNIASOBT6M36ZQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080869512 (59h56m2.803432552s) [noise=true , address=104.36.82.6:4874 , piece_id=637BK2LTKHCYE34SMXRZSYL6S436WO7DHOFSBS43M7SPQZALTZ6A, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080888517 (59h56m2.803451547s) [noise=true , address=104.36.82.13:4187 , piece_id=BDXQXYRHLV7TI326PPYYWHVM2VZWDWHZZHUSVTCIEUKD4KKYMBFQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081210333 (59h56m2.803773373s) [noise=true , address=104.36.82.4:4801 , piece_id=IGIYCYLGSAJXQ7JQMVJRKFNSNNX2JQ7EK6XIMJHXJCNHRBJXOHQA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081216144 (59h56m2.803779174s) [noise=true , address=77.90.31.10:4520 , piece_id=YUTK6OVNPAOJ66IYNBUZZVLMEUSBE46ZL34RKMWWZF5Z4ZNUQ2GQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081490639 (59h56m2.804053689s) [noise=true , address=45.91.116.14:4190 , piece_id=4OHJMBE3HJRPYMDDQD7MDFRLJR2PKBOXOLZYPCPREVF2JH24HGTA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081509566 (59h56m2.804072596s) [noise=true , address=45.91.116.7:4009 , piece_id=ZA6MQOJ4D74FY4QCDOAP52UAJRSJQRCHQG2IILYTE4SFJJNQQGPA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081732582 (59h56m2.804295623s) [noise=true , address=104.36.82.15:4259 , piece_id=7JPY57ZC4BQKU4HPKQIZ3NIH2TPQ735FK45M5F2SHINTTOJMLCUQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081804571 (59h56m2.804367611s) [noise=true , address=104.36.82.14:4221 , piece_id=VXFUFOELD4IUZER4R7ULGNFGWT6PBW4GNIL5S7CMCHOKUIBKXYEQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081986508 (59h56m2.804549548s) [noise=true , address=45.91.116.13:4154 , piece_id=37EKK7P2QHRL36Q7LEWJHNVZSJHBYLDCNU2KW4BYVL2LBWLBTBAA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:54.825954936 (59h56m4.548517976s) [upload=57499]: Next returned nil segment Jul 26 23:31:54.825973040 (59h56m4.54853608s) [upload=57499]: Waiting for error group managing segments... Jul 26 23:31:55.103202033 (59h56m4.825765063s) [noise=true , address=45.91.116.13:4154 , piece_id=37EKK7P2QHRL36Q7LEWJHNVZSJHBYLDCNU2KW4BYVL2LBWLBTBAA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.110086025 (59h56m4.832649065s) [noise=true , address=45.91.116.10:4082 , piece_id=JLVQYPCY6FEJJ54YPX5GHMMQTDYEOT6PIBAHL4DWXU2WH3TJ25VQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.112825736 (59h56m4.835388786s) [noise=true , address=45.91.116.14:4190 , piece_id=4OHJMBE3HJRPYMDDQD7MDFRLJR2PKBOXOLZYPCPREVF2JH24HGTA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.191643428 (59h56m4.914206468s) [noise=true , address=77.90.31.10:4520 , piece_id=YUTK6OVNPAOJ66IYNBUZZVLMEUSBE46ZL34RKMWWZF5Z4ZNUQ2GQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.191826358 (59h56m4.914389388s) [noise=true , address=45.91.116.8:4045 , piece_id=QXULBJBY73CKXBNNRRSQWSQKRU4L2MKLGTJVTJEF3N7LMH47AQTQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.232874405 (59h56m4.955437445s) [noise=true , address=45.91.116.7:4009 , piece_id=ZA6MQOJ4D74FY4QCDOAP52UAJRSJQRCHQG2IILYTE4SFJJNQQGPA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.244022796 (59h56m4.966585836s) [noise=true , address=45.91.116.12:4225 , piece_id=TEHDDMAKC7P2GHBJI4E34W27YZ3CRLZSZCD7B2A7FQW5MUS5SVFA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:32:22.205093270 (59h56m31.92765631s) [noise=true , address=104.36.82.12:4150 , piece_id=YIH7HSCLB7V5OARSUKSMTREKTEBTS6S4UYEUBE7PQXBLNIW77W2A, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:60362->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205155129 (59h56m31.927718169s) [noise=true , address=104.36.82.15:4259 , piece_id=7JPY57ZC4BQKU4HPKQIZ3NIH2TPQ735FK45M5F2SHINTTOJMLCUQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:57980->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205199333 (59h56m31.927762373s) [noise=true , address=104.36.82.14:4221 , piece_id=VXFUFOELD4IUZER4R7ULGNFGWT6PBW4GNIL5S7CMCHOKUIBKXYEQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:53160->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205207129 (59h56m31.927770169s) [noise=true , address=77.90.31.17:4770 , piece_id=HJFXXZV5TX2FZZF4LFUJT526D66IYHQTEJ6YTUNVRIWV4D3DACPA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:44874->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:44874->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44874->77.90.31.17:4770: use of closed network connection Jul 26 23:32:22.205240272 (59h56m31.927803312s) [noise=true , address=104.36.82.6:4874 , piece_id=637BK2LTKHCYE34SMXRZSYL6S436WO7DHOFSBS43M7SPQZALTZ6A, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205256913 (59h56m31.927819953s) [noise=true , address=104.36.82.3:4765 , piece_id=OLEWJ6R5VRNNRWVQUYX3KSG5GANPRBXW23M55OO6KSW7XQD74BRA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205285999 (59h56m31.927849039s) [noise=true , address=104.36.82.13:4187 , piece_id=BDXQXYRHLV7TI326PPYYWHVM2VZWDWHZZHUSVTCIEUKD4KKYMBFQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205290939 (59h56m31.927853989s) [noise=true , address=104.36.82.7:4910 , piece_id=X6I37G3J7KPYZVBOXGEADLTECWJRURRVVW5TKGRNIASOBT6M36ZQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205296179 (59h56m31.927859209s) [noise=true , address=104.36.82.4:4801 , piece_id=IGIYCYLGSAJXQ7JQMVJRKFNSNNX2JQ7EK6XIMJHXJCNHRBJXOHQA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205331346 (59h56m31.927894386s) [seg_pos={0 0}, upload=57499]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:32:22.440397036 (59h56m32.162960066s) [upload=57499]: Done uploading segments... Jul 26 23:32:22.440419980 (59h56m32.16298301s) [upload=57499]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.236948245s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 32 out of 60 is failed (threshold=3) [.X...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.2957381s FAIL (2025-07-26 23:32:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.295763851s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [............................XXX.X..X.....XXXXX...XXX.X.....X] duration: 30.000868758s FAIL (2025-07-26 23:32:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000891051s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235752063s FAIL (2025-07-26 23:31:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ading piece... Jul 26 23:30:54.178707139 (59h55m3.901270169s) [noise=true , address=79.127.219.38:4003 , piece_id=OIWN4NLDYBXLKAYBVHXJVDJ5PIRB5FGE6LOJR66E2B5MWJRB7UHQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.178802241 (59h55m3.901365271s) [noise=true , address=77.90.31.10:4540 , piece_id=SAJ2UXKVDCCWVG3UUV4VYR25NOVXL2I3EANBZYUH2R2HY72WRXWQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.178960754 (59h55m3.901523795s) [noise=true , address=64.49.13.140:4514 , piece_id=SVCBCM2BZDW7D556JAJX64KAFZ3LFRY6AHZ7SDMY5PS6JVRSX5RQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.179068891 (59h55m3.901631931s) [noise=true , address=104.36.82.46:4485 , piece_id=5TPDMQZPBD3ZS4PUNAXVBPKIFEF5L2LIEM4YFFHVV3DZBDJW66SQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.179209039 (59h55m3.90177208s) [noise=true , address=104.36.82.15:4276 , piece_id=F6BDHCP5PH3XFG4X7FCYHLS62JYBEX36LDYQY3ISBKOAOW64SDKA, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.179330291 (59h55m3.901893331s) [noise=true , address=104.36.82.47:4536 , piece_id=TTWORDXUOWWJRT5TQS36TPBEAN2KS4PK6WM5XG2VQL3FH2LHXDKQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.179555292 (59h55m3.902118362s) [noise=true , address=77.90.31.12:4611 , piece_id=SOXJOGSW2UKAHTC2XDGLE5REZAKGB2NDV6N4E7QTRBGTG3WG6TQA, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.180001546 (59h55m3.902564586s) [noise=true , address=104.36.82.53:4148 , piece_id=PT4RBZF62VD4Y7REHJQZWXRO4LB4EAN43FLNDG5AQAEJ3NLIQXUA, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.180532051 (59h55m3.903095101s) [noise=true , address=64.49.13.138:4482 , piece_id=2R36EPQ6IEDQHY3EE4VOM2KETVRKYKM75XN75X42GPB6Y77OZSEQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.180786066 (59h55m3.903349096s) [noise=true , address=104.36.82.18:4390 , piece_id=DUHRIT3GPWAUEVTU3L5BCQNKUFLRLAT6H5VNJTJ3NR2O3SBV7AGQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.181068067 (59h55m3.903631107s) [noise=true , address=77.90.31.11:4558 , piece_id=SM24PRN6R74SSJHWVERKYEY4G5FTLE7RCUH2L3NSNM4J6S6O5JSQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.181375244 (59h55m3.903938284s) [noise=true , address=104.36.82.55:4195 , piece_id=BZNQWSAKOTQVOLB6TZYBY4OZ3GPRZV2R5VQP33ENTYHNAVKAWJCQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.181595716 (59h55m3.904158756s) [noise=true , address=104.36.82.52:4118 , piece_id=JNVDAPVCYI7DWVWBTTRR6UI5NUSI7BLHZ5L3M6RUPGKVW4DBRGDQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.181859280 (59h55m3.90442232s) [noise=true , address=104.36.82.11:4125 , piece_id=KQJ55Y34PKKXTHG7XNCAXNXSGATNZCMJRZY2B22RX342KLREDX6Q, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:55.285566215 (59h55m5.008129255s) [upload=57488]: Next returned nil segment Jul 26 23:30:55.285584069 (59h55m5.008147109s) [upload=57488]: Waiting for error group managing segments... Jul 26 23:30:55.597217568 (59h55m5.319780608s) [noise=true , address=79.127.205.245:4245 , piece_id=HMSYQQFNUJZAWOOCH4FDKSFRHA5WBBDV2HEDIAJGXU7BGDNOT72Q, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:30:55.618511900 (59h55m5.34107494s) [noise=true , address=79.127.205.233:4655 , piece_id=TZHKBUBWCE3JBYSZ6LGMLMVUH5P7B37Z4VN6U4XPZQBQB5X62XGA, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:30:55.709907568 (59h55m5.432470599s) [noise=true , address=79.127.205.246:4282 , piece_id=AIZOQPN6BTCGUVNXL3ZO27XWLTNIHCWWX7NHMK3LNERE4LKRFTYQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:30:56.054055077 (59h55m5.776618117s) [noise=true , address=79.127.219.38:4003 , piece_id=OIWN4NLDYBXLKAYBVHXJVDJ5PIRB5FGE6LOJR66E2B5MWJRB7UHQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:30:56.159473544 (59h55m5.882036584s) [noise=true , address=77.90.31.10:4540 , piece_id=SAJ2UXKVDCCWVG3UUV4VYR25NOVXL2I3EANBZYUH2R2HY72WRXWQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:30:57.266956595 (59h55m6.989519635s) [noise=true , address=64.49.13.138:4482 , piece_id=2R36EPQ6IEDQHY3EE4VOM2KETVRKYKM75XN75X42GPB6Y77OZSEQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:31:04.113925109 (59h55m13.836488149s) [noise=true , address=64.49.13.140:4514 , piece_id=SVCBCM2BZDW7D556JAJX64KAFZ3LFRY6AHZ7SDMY5PS6JVRSX5RQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:31:49.297577029 (59h55m59.020140069s) [noise=true , address=79.127.201.215:4349 , piece_id=QLWJ4NIMF4XOUM5YKNIDA2DVME6CNIRZ27KTTH4Z7RMJA52VHJOQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:31:53.064761292 (59h56m2.787324322s) [noise=true , address=77.90.31.12:4611 , piece_id=SOXJOGSW2UKAHTC2XDGLE5REZAKGB2NDV6N4E7QTRBGTG3WG6TQA, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:1r7qdqiUuUqsKnTXdC3S1Ek84rsCPtjPoXbPugVHWBzJezv4W2, address:77.90.31.12:4611): protocol: write tcp 188.166.225.170:54474->77.90.31.12:4611: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.064807680 (59h56m2.78737072s) [noise=true , address=104.36.82.18:4390 , piece_id=DUHRIT3GPWAUEVTU3L5BCQNKUFLRLAT6H5VNJTJ3NR2O3SBV7AGQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: write tcp 188.166.225.170:50360->104.36.82.18:4390: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.064838248 (59h56m2.787401278s) [noise=true , address=104.36.82.46:4485 , piece_id=5TPDMQZPBD3ZS4PUNAXVBPKIFEF5L2LIEM4YFFHVV3DZBDJW66SQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ, address:104.36.82.46:4485): protocol: write tcp 188.166.225.170:37788->104.36.82.46:4485: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.064871302 (59h56m2.787434342s) [noise=true , address=104.36.82.15:4276 , piece_id=F6BDHCP5PH3XFG4X7FCYHLS62JYBEX36LDYQY3ISBKOAOW64SDKA, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: write tcp 188.166.225.170:39702->104.36.82.15:4276: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.064992564 (59h56m2.787555594s) [noise=true , address=104.36.82.11:4125 , piece_id=KQJ55Y34PKKXTHG7XNCAXNXSGATNZCMJRZY2B22RX342KLREDX6Q, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:4125): protocol: write tcp 188.166.225.170:47236->104.36.82.11:4125: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065023192 (59h56m2.787586232s) [noise=true , address=104.36.82.53:4148 , piece_id=PT4RBZF62VD4Y7REHJQZWXRO4LB4EAN43FLNDG5AQAEJ3NLIQXUA, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: write tcp 188.166.225.170:60288->104.36.82.53:4148: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065028081 (59h56m2.787591121s) [noise=true , address=104.36.82.52:4118 , piece_id=JNVDAPVCYI7DWVWBTTRR6UI5NUSI7BLHZ5L3M6RUPGKVW4DBRGDQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:104.36.82.52:4118): protocol: write tcp 188.166.225.170:42958->104.36.82.52:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065032661 (59h56m2.787595691s) [noise=true , address=77.90.31.11:4558 , piece_id=SM24PRN6R74SSJHWVERKYEY4G5FTLE7RCUH2L3NSNM4J6S6O5JSQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:60078->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065085050 (59h56m2.78764809s) [noise=true , address=104.36.82.55:4195 , piece_id=BZNQWSAKOTQVOLB6TZYBY4OZ3GPRZV2R5VQP33ENTYHNAVKAWJCQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: write tcp 188.166.225.170:32814->104.36.82.55:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065091874 (59h56m2.787654904s) [noise=true , address=104.36.82.47:4536 , piece_id=TTWORDXUOWWJRT5TQS36TPBEAN2KS4PK6WM5XG2VQL3FH2LHXDKQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: write tcp 188.166.225.170:37538->104.36.82.47:4536: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065103396 (59h56m2.787666436s) [seg_pos={0 0}, upload=57488]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:31:53.297872715 (59h56m3.020435755s) [upload=57488]: Done uploading segments... Jul 26 23:31:53.297901179 (59h56m3.020464219s) [upload=57488]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.23576588s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.............................XXX.X..X.....XXXXX...XXX.X.....] duration: 7.59956393s FAIL (2025-07-26 23:31:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.599577676s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [.XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..] duration: 35.883120325s FAIL (2025-07-26 23:31:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.883132749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.853814106s FAIL (2025-07-26 23:31:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.853846298s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.......X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.237683038s FAIL (2025-07-26 23:31:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 67]: Done uploading segments... Jul 26 23:30:00.101794078 (59h54m9.824357118s) [upload=57467]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Jul 26 23:30:40.084306704 (59h54m49.806869744s) [upload=57482]: Starting upload Jul 26 23:30:40.084364915 (59h54m49.806927945s) [upload=57482]: Done starting upload Jul 26 23:30:40.084446151 (59h54m49.807009191s) [upload=57482]: Uploading segments... Jul 26 23:30:40.084637487 (59h54m49.807200527s) [upload=57482]: Got next segment. Inline: false Jul 26 23:30:40.084668797 (59h54m49.807231827s) [upload=57482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:30:40.084673335 (59h54m49.807236375s) [upload=57482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:30:40.531458710 (59h54m50.254021751s) [seg_pos={0 0}, upload=57482]: Begin upload segment... Jul 26 23:30:40.531557320 (59h54m50.25412036s) [seg_pos={0 0}, upload=57482]: Done begin upload segment. Jul 26 23:30:40.531581936 (59h54m50.254144976s) [noise=true , address=77.90.31.18:4807 , piece_id=FMQJOAME6X4LLR3DCTAWKRKVZJVJQZFZ6DTLZC4CFIBBVJKOSWOQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.531590152 (59h54m50.254153192s) [noise=true , address=104.36.82.2:4729 , piece_id=R5SGBXXZLH64BCHB3NCJKJUXPP4UBQ5FPU4O5OVZA7EWM7NLIVMA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.532114545 (59h54m50.254677585s) [noise=true , address=45.91.116.13:4154 , piece_id=37LBS2PPJKGVSWSAMD6PBZSQ6QCYDU6E7PWPPJOEJL6P2IH4LW4A, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.532330970 (59h54m50.25489403s) [noise=true , address=104.36.82.7:4912 , piece_id=4CJ7X2Q3FTLXXDY3NMWABVPMCTMY5W6HSGPWDU6PQDSXTT265OSA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.532564817 (59h54m50.255127857s) [noise=true , address=45.91.116.4:4939 , piece_id=YNUMY7RKUV64QPQEYPLGJICZL42GP55ANLPWJKV4MO6CKWZP4MZQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.532979841 (59h54m50.255542901s) [noise=true , address=104.36.82.10:4079 , piece_id=JE4DDTPWHGGAEUFUDU2N6AQUF7LRIP3S3GG6SDUJBZ6A42RGPIWA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.533980686 (59h54m50.256543726s) [noise=true , address=104.36.82.9:4006 , piece_id=5SF5SAJMBPVD4JR2IFPV323POFHW6VWHI2CBRBT6L5ZIGLKLAX7Q, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.534250111 (59h54m50.256813151s) [noise=true , address=77.90.31.19:4842 , piece_id=C4TTNABYCSPPF2MWHSIHTU56GON4ZS45QZJLGNCHV6HHFGCZCMEA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.534464070 (59h54m50.25702711s) [noise=true , address=104.36.82.16:4295 , piece_id=DQSPK6R3VZSPJZSHLP7WM2GDAUYX6MUBVCDUFKVDMOVZ2MPSYA3Q, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.534684402 (59h54m50.257247432s) [noise=true , address=104.36.82.7:4911 , piece_id=RPPWFAMZIDXXA62II77JSRGZXFSZE3IHWYBDYGKD7S5AJWGRQQ4A, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.534898221 (59h54m50.257461261s) [noise=true , address=45.91.116.6:4975 , piece_id=TKKDKPFZY432JZ4QJYZKTW3HYKPI43V7XGDEBOMWRPXWYKE2HRUA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.535097552 (59h54m50.257660592s) [noise=true , address=45.91.116.12:4227 , piece_id=K3PGMRXOQQZ53HHZJJXXBDUCYMXC4NCRK3SLJRWYLXCSDHSM67XQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.535286404 (59h54m50.257849444s) [noise=true , address=45.91.116.14:4189 , piece_id=3WZXZVVWQ4CCNZGS2MLWGF5J2XUEYF35UJ2KAUV63FHWARTNZAVQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.535481337 (59h54m50.258044367s) [noise=true , address=45.91.116.5:4297 , piece_id=BIZYKQK7FDCWSCLVM2T7VYZ7VZWOXDJXBV7ALQFOGGSHECHC4BWA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.535719583 (59h54m50.258282623s) [noise=true , address=77.90.31.14:4661 , piece_id=NWG65BD3VTJFJIJKGMDWZJ3LLYQNLXMCCU3FXNOXNEYRBXZZYJIQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.535970052 (59h54m50.258533092s) [noise=true , address=77.90.31.21:4877 , piece_id=FAF25QKKR2BAW5TXDTLLO5FQVLBBIONTSKMIYPIX4ZCQCWKTBEKQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:42.267284129 (59h54m51.989847169s) [upload=57482]: Next returned nil segment Jul 26 23:30:42.267330007 (59h54m51.989893047s) [upload=57482]: Waiting for error group managing segments... Jul 26 23:30:42.604913796 (59h54m52.327476866s) [noise=true , address=45.91.116.14:4189 , piece_id=3WZXZVVWQ4CCNZGS2MLWGF5J2XUEYF35UJ2KAUV63FHWARTNZAVQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.611863303 (59h54m52.334426333s) [noise=true , address=45.91.116.4:4939 , piece_id=YNUMY7RKUV64QPQEYPLGJICZL42GP55ANLPWJKV4MO6CKWZP4MZQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.630485513 (59h54m52.353048553s) [noise=true , address=77.90.31.21:4877 , piece_id=FAF25QKKR2BAW5TXDTLLO5FQVLBBIONTSKMIYPIX4ZCQCWKTBEKQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.653457135 (59h54m52.376020175s) [noise=true , address=45.91.116.6:4975 , piece_id=TKKDKPFZY432JZ4QJYZKTW3HYKPI43V7XGDEBOMWRPXWYKE2HRUA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.703353371 (59h54m52.425916411s) [noise=true , address=45.91.116.12:4227 , piece_id=K3PGMRXOQQZ53HHZJJXXBDUCYMXC4NCRK3SLJRWYLXCSDHSM67XQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.704483914 (59h54m52.427046944s) [noise=true , address=45.91.116.13:4154 , piece_id=37LBS2PPJKGVSWSAMD6PBZSQ6QCYDU6E7PWPPJOEJL6P2IH4LW4A, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.712981372 (59h54m52.435544412s) [noise=true , address=77.90.31.14:4661 , piece_id=NWG65BD3VTJFJIJKGMDWZJ3LLYQNLXMCCU3FXNOXNEYRBXZZYJIQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:43.076229695 (59h54m52.798792725s) [noise=true , address=45.91.116.5:4297 , piece_id=BIZYKQK7FDCWSCLVM2T7VYZ7VZWOXDJXBV7ALQFOGGSHECHC4BWA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:44.199539605 (59h54m53.922102645s) [noise=true , address=77.90.31.19:4842 , piece_id=C4TTNABYCSPPF2MWHSIHTU56GON4ZS45QZJLGNCHV6HHFGCZCMEA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:31:09.442833821 (59h55m19.165396851s) [noise=true , address=104.36.82.16:4295 , piece_id=DQSPK6R3VZSPJZSHLP7WM2GDAUYX6MUBVCDUFKVDMOVZ2MPSYA3Q, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.442901150 (59h55m19.16546419s) [noise=true , address=77.90.31.18:4807 , piece_id=FMQJOAME6X4LLR3DCTAWKRKVZJVJQZFZ6DTLZC4CFIBBVJKOSWOQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:35858->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.442952097 (59h55m19.165515137s) [noise=true , address=104.36.82.10:4079 , piece_id=JE4DDTPWHGGAEUFUDU2N6AQUF7LRIP3S3GG6SDUJBZ6A42RGPIWA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:55318->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.442964661 (59h55m19.165527701s) [noise=true , address=104.36.82.7:4911 , piece_id=RPPWFAMZIDXXA62II77JSRGZXFSZE3IHWYBDYGKD7S5AJWGRQQ4A, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.442976423 (59h55m19.165539464s) [noise=true , address=104.36.82.2:4729 , piece_id=R5SGBXXZLH64BCHB3NCJKJUXPP4UBQ5FPU4O5OVZA7EWM7NLIVMA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.442996783 (59h55m19.165559823s) [noise=true , address=104.36.82.9:4006 , piece_id=5SF5SAJMBPVD4JR2IFPV323POFHW6VWHI2CBRBT6L5ZIGLKLAX7Q, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.443012263 (59h55m19.165575313s) [noise=true , address=104.36.82.7:4912 , piece_id=4CJ7X2Q3FTLXXDY3NMWABVPMCTMY5W6HSGPWDU6PQDSXTT265OSA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:54992->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.443061656 (59h55m19.165624696s) [seg_pos={0 0}, upload=57482]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:31:09.677997009 (59h55m19.400560049s) [upload=57482]: Done uploading segments... Jul 26 23:31:09.678047015 (59h55m19.400610065s) [upload=57482]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.237714849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 32 out of 60 is failed (threshold=3) [X.X...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.241701548s FAIL (2025-07-26 23:31:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.241730314s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX.] duration: 25.382920057s FAIL (2025-07-26 23:30:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.38293201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [X.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23196722s FAIL (2025-07-26 23:30:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3:29:53.250824302 (59h54m2.973387342s) [upload=57474]: Uploading segments... Jul 26 23:29:53.250999246 (59h54m2.973562287s) [upload=57474]: Got next segment. Inline: false Jul 26 23:29:53.251052939 (59h54m2.973615979s) [upload=57474]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:29:53.251057909 (59h54m2.973620949s) [upload=57474]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:29:53.707199345 (59h54m3.429762385s) [seg_pos={0 0}, upload=57474]: Begin upload segment... Jul 26 23:29:53.707275330 (59h54m3.42983837s) [seg_pos={0 0}, upload=57474]: Done begin upload segment. Jul 26 23:29:53.707320868 (59h54m3.429883948s) [noise=true , address=79.127.201.215:4351 , piece_id=E3GQBT6MVC2YLLYDLB6BBIUA2SGINRPIFY7U54H63QRELIIFRRUQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.707713709 (59h54m3.430276809s) [noise=true , address=104.36.82.52:4111 , piece_id=NN6VQF6RNUXXXP6YJEO3U24WC33PHEDQ7M66CYQ4G7YEFJ33R4EQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.708100429 (59h54m3.430663469s) [noise=true , address=104.36.82.54:4165 , piece_id=XHXFX7MUDXNVB6BGMXWCZ6YPOSQ5ZH7EHNYKLOJZIAPVMMJEF6UQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.708433466 (59h54m3.430996506s) [noise=true , address=104.36.82.46:4515 , piece_id=RMODAKQZ6OUF2N2AN5DTYL5AP3RE2PGC46VEWJLR4O4BPLLIJPKQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.708621326 (59h54m3.431184386s) [noise=true , address=79.127.205.245:4249 , piece_id=TLSRGN5GMGUUK6X3EKE4ZWSDBEKMHXM5UJNYLFCDT4Z4WHZ2KJTA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.708991523 (59h54m3.431554574s) [noise=true , address=156.146.43.225:4434 , piece_id=2YRK4ZD2CJ4WRVASOPOFE6J2NGEVXNC3YOHE2ZTJ635WHZBFDQPQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.709039015 (59h54m3.431602055s) [noise=true , address=79.127.205.235:4673 , piece_id=MLVD3LC32RVPS5RRXUGJNDSGKGHKA2UJYDTKF3NFLVG7JR2CBOZQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.709422939 (59h54m3.431985979s) [noise=true , address=79.127.205.234:4601 , piece_id=QNNWJUEHWX4VKYDK32LJCLEWEDCQCN4QP7343FO5MYWUO26KGOXQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.709466283 (59h54m3.432029313s) [noise=true , address=79.127.205.246:4272 , piece_id=AUEXTAZP7E2WEU223JAUVWBAQ37PFUKF4PLQQ6R6C3VBP3WGOPZA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.709849045 (59h54m3.432412105s) [noise=true , address=104.36.82.47:4538 , piece_id=MU2IOTANZ55QDD25IEZ7CSO5HOSRQE7DOAXOOVP5KDMDXAZGS2FA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.709928297 (59h54m3.432491337s) [noise=true , address=45.91.116.5:4307 , piece_id=4WS5QO6M4RENFGTOBNDL7BNJIKVBFBUOP7F4I6H4V5CO2JKANX4A, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.710199585 (59h54m3.432762635s) [noise=true , address=45.91.116.14:4206 , piece_id=3E6F7BFBUAMD4EQ5FIZ46T5TIPWFUY4H6Z5NLPHDHUAZRVFTAWOA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.710339664 (59h54m3.432902714s) [noise=true , address=45.91.116.14:4222 , piece_id=62DMQYPYQAQE4OWS7OB22QRHRHGHAXRE4VWZXTGBJCWAW4F2S2GA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.710503407 (59h54m3.433066447s) [noise=true , address=77.90.31.17:4792 , piece_id=CKDNLHTAZDFEIOB623NRG3DN227ND3MTDDWJ5XS5DQ23SFQPS2TA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.710686807 (59h54m3.433249858s) [noise=true , address=77.90.31.13:4639 , piece_id=CANLYTCYNGZVGDKGD4EK5XJJCBFAOIB2SYP2UBUEYRPJBTQJ3UWA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.710809413 (59h54m3.433372453s) [noise=true , address=79.127.220.97:4361 , piece_id=GVE3D45YX732VKFTQH265VYH6PMLNLK6R3OZZ2YBHTKBJ5KNFRIA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.711013202 (59h54m3.433576252s) [noise=true , address=156.146.43.225:4449 , piece_id=SEREWESX23R2IRYHSXMTOTMIBHEGNVWFV3E3VVDB43X2RWHRQFRA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.711343704 (59h54m3.433906744s) [noise=true , address=64.49.13.138:4483 , piece_id=OE5HU663BHCSNERNU6GW3LV7G5OTWNJNV6YEGZCOUWCCSNSNOE4Q, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:54.807016598 (59h54m4.529579628s) [upload=57474]: Next returned nil segment Jul 26 23:29:54.807036777 (59h54m4.529599817s) [upload=57474]: Waiting for error group managing segments... Jul 26 23:29:55.131554306 (59h54m4.854117336s) [noise=true , address=79.127.205.245:4249 , piece_id=TLSRGN5GMGUUK6X3EKE4ZWSDBEKMHXM5UJNYLFCDT4Z4WHZ2KJTA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:55.143541201 (59h54m4.866104241s) [noise=true , address=79.127.205.234:4601 , piece_id=QNNWJUEHWX4VKYDK32LJCLEWEDCQCN4QP7343FO5MYWUO26KGOXQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:55.216734031 (59h54m4.939297061s) [noise=true , address=79.127.205.246:4272 , piece_id=AUEXTAZP7E2WEU223JAUVWBAQ37PFUKF4PLQQ6R6C3VBP3WGOPZA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:55.308789230 (59h54m5.03135229s) [noise=true , address=79.127.205.235:4673 , piece_id=MLVD3LC32RVPS5RRXUGJNDSGKGHKA2UJYDTKF3NFLVG7JR2CBOZQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:55.691662282 (59h54m5.414225322s) [noise=true , address=45.91.116.14:4222 , piece_id=62DMQYPYQAQE4OWS7OB22QRHRHGHAXRE4VWZXTGBJCWAW4F2S2GA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:55.754482246 (59h54m5.477045286s) [noise=true , address=77.90.31.17:4792 , piece_id=CKDNLHTAZDFEIOB623NRG3DN227ND3MTDDWJ5XS5DQ23SFQPS2TA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:56.252467814 (59h54m5.975030844s) [noise=true , address=45.91.116.5:4307 , piece_id=4WS5QO6M4RENFGTOBNDL7BNJIKVBFBUOP7F4I6H4V5CO2JKANX4A, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:56.616698146 (59h54m6.339261176s) [noise=true , address=64.49.13.138:4483 , piece_id=OE5HU663BHCSNERNU6GW3LV7G5OTWNJNV6YEGZCOUWCCSNSNOE4Q, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:57.234672286 (59h54m6.957235386s) [noise=true , address=156.146.43.225:4434 , piece_id=2YRK4ZD2CJ4WRVASOPOFE6J2NGEVXNC3YOHE2ZTJ635WHZBFDQPQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:57.508445900 (59h54m7.23100894s) [noise=true , address=45.91.116.14:4206 , piece_id=3E6F7BFBUAMD4EQ5FIZ46T5TIPWFUY4H6Z5NLPHDHUAZRVFTAWOA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:57.678401365 (59h54m7.400964395s) [noise=true , address=156.146.43.225:4449 , piece_id=SEREWESX23R2IRYHSXMTOTMIBHEGNVWFV3E3VVDB43X2RWHRQFRA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:58.456154200 (59h54m8.17871723s) [noise=true , address=79.127.220.97:4361 , piece_id=GVE3D45YX732VKFTQH265VYH6PMLNLK6R3OZZ2YBHTKBJ5KNFRIA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:30:31.761179840 (59h54m41.483742911s) [noise=true , address=79.127.201.215:4351 , piece_id=E3GQBT6MVC2YLLYDLB6BBIUA2SGINRPIFY7U54H63QRELIIFRRUQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:30:52.604814246 (59h55m2.327377286s) [noise=true , address=77.90.31.13:4639 , piece_id=CANLYTCYNGZVGDKGD4EK5XJJCBFAOIB2SYP2UBUEYRPJBTQJ3UWA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: ecclient: upload failed (node:12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, address:77.90.31.13:4639): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:30:52.604921631 (59h55m2.327484671s) [noise=true , address=104.36.82.54:4165 , piece_id=XHXFX7MUDXNVB6BGMXWCZ6YPOSQ5ZH7EHNYKLOJZIAPVMMJEF6UQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: write tcp 188.166.225.170:59972->104.36.82.54:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:52.604958531 (59h55m2.327521571s) [noise=true , address=104.36.82.46:4515 , piece_id=RMODAKQZ6OUF2N2AN5DTYL5AP3RE2PGC46VEWJLR4O4BPLLIJPKQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 188.166.225.170:37572->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:52.604990633 (59h55m2.327553673s) [noise=true , address=104.36.82.52:4111 , piece_id=NN6VQF6RNUXXXP6YJEO3U24WC33PHEDQ7M66CYQ4G7YEFJ33R4EQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: write tcp 188.166.225.170:35610->104.36.82.52:4111: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:52.605016552 (59h55m2.327579602s) [noise=true , address=104.36.82.47:4538 , piece_id=MU2IOTANZ55QDD25IEZ7CSO5HOSRQE7DOAXOOVP5KDMDXAZGS2FA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: write tcp 188.166.225.170:46714->104.36.82.47:4538: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:52.605051108 (59h55m2.327614148s) [seg_pos={0 0}, upload=57474]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:30:52.833915595 (59h55m2.556478635s) [upload=57474]: Done uploading segments... Jul 26 23:30:52.833941204 (59h55m2.556504244s) [upload=57474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.231982218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.894924019s FAIL (2025-07-26 23:30:30 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.894949288s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX] duration: 1m0.000557355s FAIL (2025-07-26 23:30:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000580559s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................XXX.X..X.....XXXXX...XXX.X....] duration: 6.114814595s FAIL (2025-07-26 23:30:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.114838311s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...............................XXX.X..X.....XXXXX...XXX.X...] duration: 17.322098092s FAIL (2025-07-26 23:30:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.322109243s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.245970893s FAIL (2025-07-26 23:30:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ) [seg_pos={0 0}, upload=57467]: Begin upload segment... Jul 26 23:29:30.957525843 (59h53m40.680088883s) [seg_pos={0 0}, upload=57467]: Done begin upload segment. Jul 26 23:29:30.957553476 (59h53m40.680116526s) [noise=true , address=45.91.116.8:4045 , piece_id=UHUAGFWKT5D5NEELNHYYRTL6A5UHDMMZI64FJIM5CP4AR4TGF4DA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.957562202 (59h53m40.680125292s) [noise=true , address=104.36.82.16:4294 , piece_id=KR3YHDPWNGHM2Y6YKBZ2N2GZZMBTPAARC77PEXDZPBLUJ3DTEGZA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.958059744 (59h53m40.680622784s) [noise=true , address=104.36.82.4:4801 , piece_id=XNGRE732J3DX2WBVP3BVX3PSI3TR62IDCPLOFLUYACO6A4SNMSGQ, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.958425615 (59h53m40.680988665s) [noise=true , address=45.91.116.9:4262 , piece_id=M6T3HG6ESFJOLI3QKNRZNOF4RVVTSM7DHSWSOLXPNQUPIBJTTY4Q, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.958455181 (59h53m40.681018221s) [noise=true , address=104.36.82.14:4222 , piece_id=3AEM2C25R2FWVCKTN2UGSY3T2DMVKNV2PSN6Y5AF5IDKXCJA7KBQ, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.958896826 (59h53m40.681459866s) [noise=true , address=104.36.82.13:4186 , piece_id=VUJTVCKV2P5FJAQ7B5ANNVNGZYRRLTSDL4WIL5KTDLA7OR7KOYBQ, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.958912457 (59h53m40.681475497s) [noise=true , address=104.36.82.16:4293 , piece_id=AZMFGRGMIHM4Z7IQOACCH3PVK47CUA4JKWUBCN435WFGC66YE77Q, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.959368789 (59h53m40.681931829s) [noise=true , address=45.91.116.12:4227 , piece_id=SQTXQJKUOTMOMUDCTLTYRETCLJC5QIX25AZSP3H3C4BXYPBSY65A, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.959384530 (59h53m40.68194757s) [noise=true , address=77.90.31.14:4662 , piece_id=5OQJBZHAQ42HTKTT7JIT22INFSXZ7EHR4GPYVHMTMQ7EF3HD6XEA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.959800195 (59h53m40.682363235s) [noise=true , address=104.36.82.13:4187 , piece_id=LMRHEEM4U6XG4UTJSXHB3QCXQ2FFE7Z6E6LPYTE36NIXVLO26IZA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.959826545 (59h53m40.682389605s) [noise=true , address=104.36.82.2:4729 , piece_id=47JQQJTJQWVAMXIFGMPXV6SDBNMV2EW6CLH4PBTTMLNCOAUNGIOQ, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.960240377 (59h53m40.682803427s) [noise=true , address=45.91.116.4:4940 , piece_id=POXONBGOYLCBXHD7RS4DP7F4M4NAIF2YKM5HWRUJSXIOJ3KIK6XA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.960268361 (59h53m40.682831411s) [noise=true , address=104.36.82.7:4911 , piece_id=WNIVV36M26PZIJYE3VBTJXRQ3I4YFEJECGA5GLWBZZSI4RB2YJ4A, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.960600396 (59h53m40.683163436s) [noise=true , address=104.36.82.11:4114 , piece_id=ASR55YF57A5NT637BY3AEPNAVB2N44AIFCTY6VF54LZGMAIRRKBA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.960659729 (59h53m40.683222769s) [noise=true , address=104.36.82.9:4006 , piece_id=2DSHSBYLKS57RTLGXHBZ6UROCCKXUHYQEAHS7PG4AFCTSXWZE6CA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.961452526 (59h53m40.684015566s) [noise=true , address=104.36.82.2:4732 , piece_id=ZV7U43ZS4JCLCHP27Q276B2BJBXHT4CUVJ6A2NH3GURGUSIJGM4Q, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:32.727099560 (59h53m42.44966259s) [upload=57467]: Next returned nil segment Jul 26 23:29:32.727119649 (59h53m42.449682689s) [upload=57467]: Waiting for error group managing segments... Jul 26 23:29:33.056938693 (59h53m42.779501723s) [noise=true , address=45.91.116.4:4940 , piece_id=POXONBGOYLCBXHD7RS4DP7F4M4NAIF2YKM5HWRUJSXIOJ3KIK6XA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: <nil> Jul 26 23:29:33.061739307 (59h53m42.784302347s) [noise=true , address=45.91.116.8:4045 , piece_id=UHUAGFWKT5D5NEELNHYYRTL6A5UHDMMZI64FJIM5CP4AR4TGF4DA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: <nil> Jul 26 23:29:33.070898932 (59h53m42.793461962s) [noise=true , address=45.91.116.9:4262 , piece_id=M6T3HG6ESFJOLI3QKNRZNOF4RVVTSM7DHSWSOLXPNQUPIBJTTY4Q, seg_pos={0 0}, upload=57467]: Done uploading piece. err: <nil> Jul 26 23:29:33.071978769 (59h53m42.794541809s) [noise=true , address=45.91.116.12:4227 , piece_id=SQTXQJKUOTMOMUDCTLTYRETCLJC5QIX25AZSP3H3C4BXYPBSY65A, seg_pos={0 0}, upload=57467]: Done uploading piece. err: <nil> Jul 26 23:29:33.203596053 (59h53m42.926159093s) [noise=true , address=77.90.31.14:4662 , piece_id=5OQJBZHAQ42HTKTT7JIT22INFSXZ7EHR4GPYVHMTMQ7EF3HD6XEA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: <nil> Jul 26 23:29:59.857576824 (59h54m9.580139864s) [noise=true , address=104.36.82.16:4293 , piece_id=AZMFGRGMIHM4Z7IQOACCH3PVK47CUA4JKWUBCN435WFGC66YE77Q, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.857761588 (59h54m9.580324628s) [noise=true , address=104.36.82.16:4294 , piece_id=KR3YHDPWNGHM2Y6YKBZ2N2GZZMBTPAARC77PEXDZPBLUJ3DTEGZA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.857904701 (59h54m9.580467741s) [noise=true , address=104.36.82.2:4729 , piece_id=47JQQJTJQWVAMXIFGMPXV6SDBNMV2EW6CLH4PBTTMLNCOAUNGIOQ, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.857991577 (59h54m9.580554687s) [noise=true , address=104.36.82.14:4222 , piece_id=3AEM2C25R2FWVCKTN2UGSY3T2DMVKNV2PSN6Y5AF5IDKXCJA7KBQ, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858028538 (59h54m9.580591578s) [noise=true , address=104.36.82.13:4186 , piece_id=VUJTVCKV2P5FJAQ7B5ANNVNGZYRRLTSDL4WIL5KTDLA7OR7KOYBQ, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858061822 (59h54m9.580624862s) [noise=true , address=104.36.82.7:4911 , piece_id=WNIVV36M26PZIJYE3VBTJXRQ3I4YFEJECGA5GLWBZZSI4RB2YJ4A, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858151223 (59h54m9.580714263s) [noise=true , address=104.36.82.13:4187 , piece_id=LMRHEEM4U6XG4UTJSXHB3QCXQ2FFE7Z6E6LPYTE36NIXVLO26IZA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858177193 (59h54m9.580740233s) [noise=true , address=104.36.82.11:4114 , piece_id=ASR55YF57A5NT637BY3AEPNAVB2N44AIFCTY6VF54LZGMAIRRKBA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858210025 (59h54m9.580773065s) [noise=true , address=104.36.82.2:4732 , piece_id=ZV7U43ZS4JCLCHP27Q276B2BJBXHT4CUVJ6A2NH3GURGUSIJGM4Q, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:38716->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858221357 (59h54m9.580784397s) [noise=true , address=104.36.82.4:4801 , piece_id=XNGRE732J3DX2WBVP3BVX3PSI3TR62IDCPLOFLUYACO6A4SNMSGQ, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858241405 (59h54m9.580804445s) [noise=true , address=104.36.82.9:4006 , piece_id=2DSHSBYLKS57RTLGXHBZ6UROCCKXUHYQEAHS7PG4AFCTSXWZE6CA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858260843 (59h54m9.580823883s) [seg_pos={0 0}, upload=57467]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:30:00.101767948 (59h54m9.824330978s) [upload=57467]: Done uploading segments... Jul 26 23:30:00.101794078 (59h54m9.824357118s) [upload=57467]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.245994508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [.X.X...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.245926377s FAIL (2025-07-26 23:29:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.245947015s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.258034307s FAIL (2025-07-26 23:29:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== , piece_id=FCF44UOI6KB6YYOXAJFQXYKHURIHBHV4KU4H63UWKW2R357UO7JA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.201952808 (59h53m2.924515848s) [noise=true , address=79.127.205.233:4633 , piece_id=LIXBVOW3ZT3MMN5JJUATCD4ACWIYHU5CTV46LBCRA4AQYXP5ALNQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.202218657 (59h53m2.924781697s) [noise=true , address=45.91.116.14:4194 , piece_id=JZNRF5IGDRMUPLGGUKNSCMD3HXERTD37KOZKKZC26OTUOGEIMHNQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.202520945 (59h53m2.925083985s) [noise=true , address=45.91.116.13:4175 , piece_id=37R4ZOXVNUPKA5TWCGUQIY25UUTDRRERXQ2VKHF5XRH5LC4OOZGQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.202767176 (59h53m2.925330206s) [noise=true , address=104.36.82.46:4489 , piece_id=VGLV6UBB5TUPIPVVHYE4LUQ7MTVM3RMWARUN4X2N3OUARQ3EIJEQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.202970264 (59h53m2.925533304s) [noise=true , address=79.127.219.38:4006 , piece_id=BQPEWDHZXHPSEKB5IDJMDLSBSEPPTMTZ6UBTQXWLN3G75RVOLLLA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.203170648 (59h53m2.925733689s) [noise=true , address=45.91.116.14:4206 , piece_id=IIJ5RBS253DUUXS4HMIE4FFVBAGGEEOBAV3MRK5APZEYPCYLUALQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.203371442 (59h53m2.925934482s) [noise=true , address=79.127.205.245:4242 , piece_id=R4N52O3SW5H4SU45G4HTI4GJ36HTNJ5AADX2G4A5EW4SG5O2QR3Q, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.203622172 (59h53m2.926185212s) [noise=true , address=104.36.82.47:4546 , piece_id=4L2HVKDNU6YOHWOSW5EPOFTDMSBBSIZBIDRFHFXYHK4NOLB4A6QA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.203816554 (59h53m2.926379594s) [noise=true , address=104.36.82.55:4197 , piece_id=RFYPIJTOHPDSSUATKJQND3GAY43FNPKMPOTFKI23BHLIFVMW26IQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.203999103 (59h53m2.926562143s) [noise=true , address=104.36.82.9:4027 , piece_id=QRZNQU47JNKE42REOQTOPEKLTT3RMQJKG42ADW4AK5DKMPCQ7INA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.204212271 (59h53m2.926775311s) [noise=true , address=156.146.43.225:4448 , piece_id=67ITBWULB6ZTXA3XSSUBWKUK3P7G6SQLEWAYPGUEOCDMMXLQPIKQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.204395862 (59h53m2.926958902s) [noise=true , address=104.36.82.53:4128 , piece_id=ATYIQZITUQIFVWMC7SNA4F56Q2J7PTBZFCI4H42OM4X3HXAZMGPQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.204578883 (59h53m2.927141923s) [noise=true , address=45.91.116.14:4222 , piece_id=NRYKVAKFCPFQUNY6B5XIDT73E2NFSKX3DQKRCGNZRO3S44Q3LRFQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.204781961 (59h53m2.927345001s) [noise=true , address=79.127.220.101:4041 , piece_id=Q5QQT6AI23Y66IN5K45M2R7WMWPWWGWG5CA5J4M6XSCB2BDXVQOA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.204957988 (59h53m2.927521028s) [noise=true , address=104.36.82.54:4162 , piece_id=4VJDYSUKPOOMIAFARIWUNBT4AZZDQ76WA6776H52I3VYBFKV5VCA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.209476623 (59h53m2.932039663s) [noise=true , address=77.90.31.16:4764 , piece_id=CDH3FO6MBSSJO67TBUZ45YRW5WUUIATXAAGQLIKTMP3QXRL3RGOA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:54.297433341 (59h53m4.019996381s) [upload=57456]: Next returned nil segment Jul 26 23:28:54.297452078 (59h53m4.020015118s) [upload=57456]: Waiting for error group managing segments... Jul 26 23:28:54.648486091 (59h53m4.371049121s) [noise=true , address=79.127.205.245:4242 , piece_id=R4N52O3SW5H4SU45G4HTI4GJ36HTNJ5AADX2G4A5EW4SG5O2QR3Q, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:54.748331225 (59h53m4.470894265s) [noise=true , address=79.127.205.233:4633 , piece_id=LIXBVOW3ZT3MMN5JJUATCD4ACWIYHU5CTV46LBCRA4AQYXP5ALNQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:55.064117540 (59h53m4.78668058s) [noise=true , address=79.127.219.38:4006 , piece_id=BQPEWDHZXHPSEKB5IDJMDLSBSEPPTMTZ6UBTQXWLN3G75RVOLLLA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:55.207205590 (59h53m4.929768641s) [noise=true , address=77.90.31.17:4787 , piece_id=FCF44UOI6KB6YYOXAJFQXYKHURIHBHV4KU4H63UWKW2R357UO7JA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:55.743367591 (59h53m5.465930621s) [noise=true , address=45.91.116.14:4222 , piece_id=NRYKVAKFCPFQUNY6B5XIDT73E2NFSKX3DQKRCGNZRO3S44Q3LRFQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 23:28:56.887766172 (59h53m6.610329212s) [noise=true , address=156.146.43.225:4448 , piece_id=67ITBWULB6ZTXA3XSSUBWKUK3P7G6SQLEWAYPGUEOCDMMXLQPIKQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:57.151823643 (59h53m6.874386683s) [noise=true , address=45.91.116.14:4194 , piece_id=JZNRF5IGDRMUPLGGUKNSCMD3HXERTD37KOZKKZC26OTUOGEIMHNQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:57.929273047 (59h53m7.651836127s) [noise=true , address=45.91.116.13:4175 , piece_id=37R4ZOXVNUPKA5TWCGUQIY25UUTDRRERXQ2VKHF5XRH5LC4OOZGQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:58.070012284 (59h53m7.792575314s) [noise=true , address=45.91.116.14:4206 , piece_id=IIJ5RBS253DUUXS4HMIE4FFVBAGGEEOBAV3MRK5APZEYPCYLUALQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:28:58.184864451 (59h53m7.907427491s) [noise=true , address=79.127.220.101:4041 , piece_id=Q5QQT6AI23Y66IN5K45M2R7WMWPWWGWG5CA5J4M6XSCB2BDXVQOA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:29:03.615820933 (59h53m13.338384003s) [noise=true , address=77.90.31.13:4634 , piece_id=C6ECRJ6O63TCTDOAB4ICSGTY3LR2Q433MNJFRYEWWNG3255K6CUQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:29:52.120011425 (59h54m1.842574465s) [noise=true , address=104.36.82.46:4489 , piece_id=VGLV6UBB5TUPIPVVHYE4LUQ7MTVM3RMWARUN4X2N3OUARQ3EIJEQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: write tcp 188.166.225.170:54066->104.36.82.46:4489: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120100366 (59h54m1.842663406s) [noise=true , address=104.36.82.47:4546 , piece_id=4L2HVKDNU6YOHWOSW5EPOFTDMSBBSIZBIDRFHFXYHK4NOLB4A6QA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: write tcp 188.166.225.170:60352->104.36.82.47:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120153206 (59h54m1.842716246s) [noise=true , address=104.36.82.55:4197 , piece_id=RFYPIJTOHPDSSUATKJQND3GAY43FNPKMPOTFKI23BHLIFVMW26IQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: write tcp 188.166.225.170:33082->104.36.82.55:4197: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120250443 (59h54m1.842813483s) [noise=true , address=77.90.31.16:4764 , piece_id=CDH3FO6MBSSJO67TBUZ45YRW5WUUIATXAAGQLIKTMP3QXRL3RGOA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, address:77.90.31.16:4764): protocol: write tcp 188.166.225.170:53384->77.90.31.16:4764: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120321940 (59h54m1.84288497s) [noise=true , address=104.36.82.9:4027 , piece_id=QRZNQU47JNKE42REOQTOPEKLTT3RMQJKG42ADW4AK5DKMPCQ7INA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:1SWk13XvkrCYwKcdZ46SK2THQvhsD6pf4hWJKgARztB9qwfMZU, address:104.36.82.9:4027): protocol: write tcp 188.166.225.170:43060->104.36.82.9:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120342729 (59h54m1.842905769s) [noise=true , address=104.36.82.53:4128 , piece_id=ATYIQZITUQIFVWMC7SNA4F56Q2J7PTBZFCI4H42OM4X3HXAZMGPQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 188.166.225.170:47254->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120396602 (59h54m1.842959642s) [noise=true , address=104.36.82.54:4162 , piece_id=4VJDYSUKPOOMIAFARIWUNBT4AZZDQ76WA6776H52I3VYBFKV5VCA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: rpc: dial tcp 104.36.82.54:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120416951 (59h54m1.842979991s) [seg_pos={0 0}, upload=57456]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:29:52.375798713 (59h54m2.098361753s) [upload=57456]: Done uploading segments... Jul 26 23:29:52.375820325 (59h54m2.098383365s) [upload=57456]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.258050348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.875904769s FAIL (2025-07-26 23:29:52 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.875930078s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.828177846s FAIL (2025-07-26 23:29:28 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.828202903s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.229738813s FAIL (2025-07-26 23:29:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== RJ2SZPA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.385005830 (59h53m6.10756886s) [noise=true , address=77.90.31.13:4626 , piece_id=YPPZQ45IRVUZR75V3NL4BPEHJBRDW7EVGTZTFDWUSOCFZC6ZPRJQ, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.385329399 (59h53m6.107892439s) [noise=true , address=104.36.82.4:4801 , piece_id=LDYPYGMJM6PUP6ROZVF5BEDMFZEVBZZUCP6X77GHFX2JK2ALJ3HA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.385336202 (59h53m6.107899232s) [noise=true , address=104.36.82.7:4909 , piece_id=4DVNFWI5FXUUWHBGKRPP6KPW4AEG3QWIYDI75KI4KUOVSX2KI26Q, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.385993239 (59h53m6.108556279s) [noise=true , address=104.36.82.5:4839 , piece_id=NF3EMKQNTW3MZEQ4PI3GBSUSB7JV3737IGB24FLQYSHHERPUQJNA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.386355362 (59h53m6.108918402s) [noise=true , address=104.36.82.12:4150 , piece_id=LQPJI7TBNZKNQAUOV3JE6PXC4KCOD6WOWYR3GC7CSNAZL2J64O3A, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.386599499 (59h53m6.109162539s) [noise=true , address=104.36.82.13:4185 , piece_id=2XWX753JZZWFBFGPYHKXPSM3GVKRSEO2M5CHS75Z4J3K35QCCOSQ, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.387006628 (59h53m6.109569668s) [noise=true , address=45.91.116.6:4974 , piece_id=5UCBKIHHG56VB4HRGVKZZSNHK2UWATQ3JD5L3I5VLG5YJXGSQDSQ, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.387276013 (59h53m6.109839073s) [noise=true , address=104.36.82.10:4079 , piece_id=XDDXUCTOM7UDYO7SX5VEHFSML6O2WJPAO5SGNELCIRB7RVIAH47A, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.387658394 (59h53m6.110221434s) [noise=true , address=104.36.82.16:4293 , piece_id=KDBNFAZRNA53O2SHLEMCZ7VFR5T32WJIN3WUUPWM2FI4Z2QHJRQA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.387900097 (59h53m6.110463137s) [noise=true , address=77.90.31.10:4518 , piece_id=GPND4PW3X6STFXHVT24XHTP3JMVBDZP6JZGGXW4B6Z4LPZ3K3XFQ, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.388186605 (59h53m6.110749655s) [noise=true , address=104.36.82.9:4005 , piece_id=UZHDIVV6LSRQLYZ7S7643D4CJZ36WVTWBYRPFLWTTWXCE6A2WZ3Q, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.388387620 (59h53m6.11095066s) [noise=true , address=104.36.82.11:4113 , piece_id=YZ2TYDLKHX25JU5ZYOMHPNAEBXIKRAS4KR45GTFZS3S32YBFUPLA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.388601920 (59h53m6.11116496s) [noise=true , address=77.90.31.16:4733 , piece_id=X6JP7VU4IIZKCMOKCNUKB65UTPUXVTQU7JLPB2OMUFL7XYO6UVNA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:58.202080039 (59h53m7.924643079s) [upload=57458]: Next returned nil segment Jul 26 23:28:58.202096922 (59h53m7.924659962s) [upload=57458]: Waiting for error group managing segments... Jul 26 23:28:58.509145339 (59h53m8.231708369s) [noise=true , address=45.91.116.6:4974 , piece_id=5UCBKIHHG56VB4HRGVKZZSNHK2UWATQ3JD5L3I5VLG5YJXGSQDSQ, seg_pos={0 0}, upload=57458]: Done uploading piece. err: <nil> Jul 26 23:28:58.587316152 (59h53m8.309879192s) [noise=true , address=77.90.31.10:4518 , piece_id=GPND4PW3X6STFXHVT24XHTP3JMVBDZP6JZGGXW4B6Z4LPZ3K3XFQ, seg_pos={0 0}, upload=57458]: Done uploading piece. err: <nil> Jul 26 23:29:00.003774366 (59h53m9.726337416s) [noise=true , address=77.90.31.13:4626 , piece_id=YPPZQ45IRVUZR75V3NL4BPEHJBRDW7EVGTZTFDWUSOCFZC6ZPRJQ, seg_pos={0 0}, upload=57458]: Done uploading piece. err: <nil> Jul 26 23:29:25.514764348 (59h53m35.237327378s) [noise=true , address=104.36.82.9:4005 , piece_id=UZHDIVV6LSRQLYZ7S7643D4CJZ36WVTWBYRPFLWTTWXCE6A2WZ3Q, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:45048->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.514822198 (59h53m35.237385238s) [noise=true , address=104.36.82.3:4765 , piece_id=BEOG7KHUHS74JKMFI6EUXM6UKGB5M2YII4AGTDURMHRTPVEHNSZA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.514862476 (59h53m35.237425516s) [noise=true , address=104.36.82.2:4729 , piece_id=ZNO2MUPZHYH3DGX7XM35IJITSWY6DBT3HVCN3OZ3IOU4BRJ2SZPA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:34642->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.514894948 (59h53m35.237457988s) [noise=true , address=104.36.82.4:4801 , piece_id=LDYPYGMJM6PUP6ROZVF5BEDMFZEVBZZUCP6X77GHFX2JK2ALJ3HA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.514928131 (59h53m35.237491171s) [noise=true , address=104.36.82.10:4077 , piece_id=CXNP6AZHBEAJBZHSXW6BVAVUJMDULEFLVLG3W6GI2SJLDRIU5PEA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:41600->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.514962617 (59h53m35.237525657s) [noise=true , address=104.36.82.7:4909 , piece_id=4DVNFWI5FXUUWHBGKRPP6KPW4AEG3QWIYDI75KI4KUOVSX2KI26Q, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:36100->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515017141 (59h53m35.237580181s) [noise=true , address=104.36.82.5:4839 , piece_id=NF3EMKQNTW3MZEQ4PI3GBSUSB7JV3737IGB24FLQYSHHERPUQJNA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:53076->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515051326 (59h53m35.237614366s) [noise=true , address=104.36.82.12:4150 , piece_id=LQPJI7TBNZKNQAUOV3JE6PXC4KCOD6WOWYR3GC7CSNAZL2J64O3A, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515098807 (59h53m35.237661837s) [noise=true , address=104.36.82.16:4293 , piece_id=KDBNFAZRNA53O2SHLEMCZ7VFR5T32WJIN3WUUPWM2FI4Z2QHJRQA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:44688->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515129135 (59h53m35.237692205s) [noise=true , address=104.36.82.11:4113 , piece_id=YZ2TYDLKHX25JU5ZYOMHPNAEBXIKRAS4KR45GTFZS3S32YBFUPLA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:52024->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515170936 (59h53m35.237733976s) [noise=true , address=77.90.31.16:4733 , piece_id=X6JP7VU4IIZKCMOKCNUKB65UTPUXVTQU7JLPB2OMUFL7XYO6UVNA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515203047 (59h53m35.237766077s) [noise=true , address=104.36.82.13:4185 , piece_id=2XWX753JZZWFBFGPYHKXPSM3GVKRSEO2M5CHS75Z4J3K35QCCOSQ, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515304642 (59h53m35.237867722s) [noise=true , address=104.36.82.10:4079 , piece_id=XDDXUCTOM7UDYO7SX5VEHFSML6O2WJPAO5SGNELCIRB7RVIAH47A, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:48046->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515338236 (59h53m35.237901276s) [seg_pos={0 0}, upload=57458]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Jul 26 23:29:25.741941816 (59h53m35.464504846s) [upload=57458]: Done uploading segments... Jul 26 23:29:25.741968177 (59h53m35.464531217s) [upload=57458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 30.229760525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [.X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XX] duration: 1m0.000458976s FAIL (2025-07-26 23:29:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000472501s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 30 out of 60 is failed (threshold=3) [..X.X...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.241411355s FAIL (2025-07-26 23:29:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.241423428s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [................................XXX.X..X.....XXXXX...XXX.X..] duration: 6.313080419s FAIL (2025-07-26 23:29:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.313099315s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.866366663s FAIL (2025-07-26 23:28:54 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.866393526s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.229944077s FAIL (2025-07-26 23:28:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.763840970 (59h52m2.48640401s) [noise=true , address=104.36.82.54:4163 , piece_id=SBIK3P3TIVW5J2FJPBY43DHTTWJK64F4Q3FYE7TGRPHKN5TVUCCA, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.764173095 (59h52m2.486736125s) [noise=true , address=77.90.31.11:4585 , piece_id=Z2KJZ6FUT2UV7HDCPVCYZNM7K6RUWHADTDQOPWKR3WSRCYRZXY7A, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.764510591 (59h52m2.487073621s) [noise=true , address=79.127.201.215:4343 , piece_id=IPEOHO2NYUP6XBI6SRYF2HOSHEMXJSS4VQDYCFB6CTDUWVPUWM4A, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.764638366 (59h52m2.487201406s) [noise=true , address=79.127.205.234:4619 , piece_id=AE3VKPOEB5XAJQ3URPQYNW5UH6HZ7QKVE5RDYHIDQLW3YW7QIGNQ, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.764785396 (59h52m2.487348437s) [noise=true , address=45.91.116.12:4230 , piece_id=3G2NIWYVZAZKLG4YYT5JES4ZB2DCMYMGKYDZLZCBPVAX2PQFVJ7A, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.764996691 (59h52m2.487559731s) [noise=true , address=79.127.201.215:4357 , piece_id=PIE5NDJWI6VBOTX4DTNL2NMDQ7LSSRPDUAPOKIDU5QGND4OUAVYQ, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.765052227 (59h52m2.487615267s) [noise=true , address=104.36.82.47:4550 , piece_id=5HQTPLSZYBGHLBTY5PXVN6MT4LYN66XZ63XNPRPGNHYLVEHHJ5SQ, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.765355838 (59h52m2.487918878s) [noise=true , address=64.49.13.138:4483 , piece_id=PHKYNVXNXZDYIGFBGU3EDTFDQGKD747NBLDFQCNNHFXOUYWIMXBQ, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.765372449 (59h52m2.487935489s) [noise=true , address=104.36.82.46:4498 , piece_id=N3BLD3IIQTLOFA7YUETGJNOJKTYMVTKUX6BVXCDZZOPL53QE5OEA, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.765634301 (59h52m2.488197341s) [noise=true , address=104.36.82.53:4124 , piece_id=7CIM7SRNRTFJ6BP2MHEFIRMBWKA4US2P2D3KRWBITT7DFFMMT3ZQ, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.765881123 (59h52m2.488444153s) [noise=true , address=104.36.82.56:4240 , piece_id=DCDDVLNN2YYZZUZZFXNXIJSUVZ6KEPBCFCNGFR7MN6QDXSRAQFYQ, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.766083179 (59h52m2.488646219s) [noise=true , address=45.91.116.14:4194 , piece_id=LMLKLE2LIUEW3JCBAMBZSZI6LIJ7QJ4AKTOJS3GYA3O7EOB3A5PQ, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.766278222 (59h52m2.488841252s) [noise=true , address=64.49.13.138:4482 , piece_id=4WYDL7OLZ54BBNM5VSSPG6HGWSRPTMVYXRY3K2SZLFFM2F6QVGOQ, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.766539172 (59h52m2.489102212s) [noise=true , address=64.49.13.137:4502 , piece_id=D5SDEUWWRVSVN7WTTLKXTAHYZWS4ENKD7UWC4FL4IRARWMDNQD6Q, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.766900614 (59h52m2.489463654s) [noise=true , address=104.36.82.52:4093 , piece_id=SGG3T4HTW66BGLCQR4USS5SRYGYFUWY7GSIFOVK3ZCTEGC54U2HQ, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.767155150 (59h52m2.48971819s) [noise=true , address=104.36.82.55:4226 , piece_id=V22QGW7PK6HKT4IUTPLI3PHFKKD72BHPMBMVDFA7EJ6CFCJWMSMQ, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:52.767417272 (59h52m2.489980312s) [noise=true , address=45.91.116.13:4175 , piece_id=YZ2IYMOQA47QJTJ4XIYVNEQBANBJMRJ3Q4OXL5FVNOITYF5ROTTQ, seg_pos={0 0}, upload=57440]: Uploading piece... Jul 26 23:27:53.885400571 (59h52m3.607963611s) [upload=57440]: Next returned nil segment Jul 26 23:27:53.885418746 (59h52m3.607981786s) [upload=57440]: Waiting for error group managing segments... Jul 26 23:27:54.169260211 (59h52m3.891823251s) [noise=true , address=79.127.205.234:4619 , piece_id=AE3VKPOEB5XAJQ3URPQYNW5UH6HZ7QKVE5RDYHIDQLW3YW7QIGNQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: <nil> Jul 26 23:27:54.215669776 (59h52m3.938232816s) [noise=true , address=64.49.13.137:4502 , piece_id=D5SDEUWWRVSVN7WTTLKXTAHYZWS4ENKD7UWC4FL4IRARWMDNQD6Q, seg_pos={0 0}, upload=57440]: Done uploading piece. err: <nil> Jul 26 23:27:54.751043938 (59h52m4.473606968s) [noise=true , address=45.91.116.13:4175 , piece_id=YZ2IYMOQA47QJTJ4XIYVNEQBANBJMRJ3Q4OXL5FVNOITYF5ROTTQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: <nil> Jul 26 23:27:55.451962848 (59h52m5.174525878s) [noise=true , address=64.49.13.138:4483 , piece_id=PHKYNVXNXZDYIGFBGU3EDTFDQGKD747NBLDFQCNNHFXOUYWIMXBQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 23:27:59.316748660 (59h52m9.0393117s) [noise=true , address=64.49.13.138:4482 , piece_id=4WYDL7OLZ54BBNM5VSSPG6HGWSRPTMVYXRY3K2SZLFFM2F6QVGOQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: <nil> Jul 26 23:27:59.736154860 (59h52m9.4587179s) [noise=true , address=79.127.220.101:4048 , piece_id=S6J3YAAWRP7TOBHWJPAOLX4NQZVGMFKRB3PXYSPDDGOTVAGUNLLQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: <nil> Jul 26 23:28:03.481186474 (59h52m13.203749514s) [noise=true , address=45.91.116.12:4230 , piece_id=3G2NIWYVZAZKLG4YYT5JES4ZB2DCMYMGKYDZLZCBPVAX2PQFVJ7A, seg_pos={0 0}, upload=57440]: Done uploading piece. err: <nil> Jul 26 23:28:08.230552986 (59h52m17.953116026s) [noise=true , address=79.127.201.215:4343 , piece_id=IPEOHO2NYUP6XBI6SRYF2HOSHEMXJSS4VQDYCFB6CTDUWVPUWM4A, seg_pos={0 0}, upload=57440]: Done uploading piece. err: <nil> Jul 26 23:28:08.502178327 (59h52m18.224741377s) [noise=true , address=45.91.116.14:4194 , piece_id=LMLKLE2LIUEW3JCBAMBZSZI6LIJ7QJ4AKTOJS3GYA3O7EOB3A5PQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: <nil> Jul 26 23:28:38.858125906 (59h52m48.580688936s) [noise=true , address=79.127.201.215:4357 , piece_id=PIE5NDJWI6VBOTX4DTNL2NMDQ7LSSRPDUAPOKIDU5QGND4OUAVYQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: <nil> Jul 26 23:28:51.668161250 (59h53m1.39072429s) [noise=true , address=77.90.31.11:4585 , piece_id=Z2KJZ6FUT2UV7HDCPVCYZNM7K6RUWHADTDQOPWKR3WSRCYRZXY7A, seg_pos={0 0}, upload=57440]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: rpc: dial tcp 77.90.31.11:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:51.668237336 (59h53m1.390800366s) [noise=true , address=104.36.82.53:4124 , piece_id=7CIM7SRNRTFJ6BP2MHEFIRMBWKA4US2P2D3KRWBITT7DFFMMT3ZQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: rpc: dial tcp 104.36.82.53:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:51.668244119 (59h53m1.390807149s) [noise=true , address=104.36.82.47:4550 , piece_id=5HQTPLSZYBGHLBTY5PXVN6MT4LYN66XZ63XNPRPGNHYLVEHHJ5SQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: ecclient: upload failed (node:1Fz8JANjsHWJkHj7Y2JAkr7NTKqDtcKjRW4kDaYGLJYwgWnxbS, address:104.36.82.47:4550): protocol: rpc: dial tcp 104.36.82.47:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:51.668278305 (59h53m1.390841345s) [noise=true , address=104.36.82.56:4240 , piece_id=DCDDVLNN2YYZZUZZFXNXIJSUVZ6KEPBCFCNGFR7MN6QDXSRAQFYQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: ecclient: upload failed (node:129QqkFEzZdtzQ6HAKxPWxDNgDVUNJJLH59oRVXnHExrwkDECyf, address:104.36.82.56:4240): protocol: rpc: dial tcp 104.36.82.56:4240: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:51.668298052 (59h53m1.390861092s) [noise=true , address=104.36.82.46:4498 , piece_id=N3BLD3IIQTLOFA7YUETGJNOJKTYMVTKUX6BVXCDZZOPL53QE5OEA, seg_pos={0 0}, upload=57440]: Done uploading piece. err: ecclient: upload failed (node:1jyjxdWkBRBkDHSKknNmPdcnJ3HRcSCUQNnmpb1CaiEdMPEGk4, address:104.36.82.46:4498): protocol: rpc: dial tcp 104.36.82.46:4498: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:51.668324342 (59h53m1.390887372s) [noise=true , address=104.36.82.55:4226 , piece_id=V22QGW7PK6HKT4IUTPLI3PHFKKD72BHPMBMVDFA7EJ6CFCJWMSMQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: ecclient: upload failed (node:1qN8jYqiZGuPjqHSp6siqdT4p33qvf1vYdAkMZKRnFgcQj9Ga3, address:104.36.82.55:4226): protocol: write tcp 188.166.225.170:56716->104.36.82.55:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:51.668338831 (59h53m1.390901861s) [noise=true , address=104.36.82.52:4093 , piece_id=SGG3T4HTW66BGLCQR4USS5SRYGYFUWY7GSIFOVK3ZCTEGC54U2HQ, seg_pos={0 0}, upload=57440]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: rpc: dial tcp 104.36.82.52:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:51.668365201 (59h53m1.390928241s) [noise=true , address=104.36.82.54:4163 , piece_id=SBIK3P3TIVW5J2FJPBY43DHTTWJK64F4Q3FYE7TGRPHKN5TVUCCA, seg_pos={0 0}, upload=57440]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: rpc: dial tcp 104.36.82.54:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:51.668388766 (59h53m1.390951806s) [seg_pos={0 0}, upload=57440]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:28:51.895793568 (59h53m1.618356598s) [upload=57440]: Done uploading segments... Jul 26 23:28:51.895817784 (59h53m1.618380824s) [upload=57440]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.229956442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 29 out of 60 is failed (threshold=3) [...X.X...........................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.231977848s FAIL (2025-07-26 23:28:41 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.232006223s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.228715696s FAIL (2025-07-26 23:28:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 3609645s) [noise=true , address=104.36.82.2:4731 , piece_id=LFWL6CQKC6WV7XQ5M2JB7LIDMRTP4IAUDPDCBNA6POUWFR5LYGGA, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.041362400 (59h52m19.76392543s) [noise=true , address=104.36.82.4:4801 , piece_id=FMGQRE2T4FN63ILTVROVJL2YISPAAB6TJXFN2X2E6WBBADCHPX5Q, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.041441341 (59h52m19.764004381s) [noise=true , address=104.36.82.9:4005 , piece_id=N66CUYLW5YR4GK6YCEFH72ZW66TDLHYJ6HIKZMLYZH7JSAC5LZZA, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.041633930 (59h52m19.76419696s) [noise=true , address=104.36.82.4:4804 , piece_id=LZG6ICH37TMXGQ4TFZT5POT7CV3LI7YSABTXQ25UTQJSH3T3PAUQ, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.041908655 (59h52m19.764471695s) [noise=true , address=104.36.82.15:4257 , piece_id=KI45UZJOTRLINZ77JN7ND5S64VUEFWLRKDZUE3CXHE6QNQQQFTSQ, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.042159886 (59h52m19.764722926s) [noise=true , address=45.91.116.8:4045 , piece_id=DJCCE5JKJ434XETX4IXNPB4JZJ6CLGRTCJX2ZO6TGFDFKDDMJ6GQ, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.042402630 (59h52m19.76496567s) [noise=true , address=104.36.82.6:4874 , piece_id=2F2GVGVCGTF3LI6NB3NYFMPOLH43U4UCSOP3TNCYJ5UPAXT6OLAQ, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.042664761 (59h52m19.765227801s) [noise=true , address=45.91.116.13:4154 , piece_id=FBQ2LUSOZTWNMYKQUUNYPZL4LFF3RNHKL3F3AIYDH7QVPMVR72JA, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.042851238 (59h52m19.765414278s) [noise=true , address=45.91.116.7:4009 , piece_id=FIXO7FBDDMTKTTHQTE77P5DHO4XAKIRQE5EWQDFJ4ZCXS2DBCOAQ, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.043049107 (59h52m19.765612147s) [noise=true , address=104.36.82.5:4839 , piece_id=EAHM5GORS2UK53HXLFJGGJPKE4IDRDTXRM5VU2CMB7OGTODAL5CQ, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.043231666 (59h52m19.765794706s) [noise=true , address=45.91.116.12:4225 , piece_id=JV7A7QI3XQQYIXD3RYC5Q5H7NRMIDSGQIK7O2B5UZL7I6JJ6HWMA, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.043413394 (59h52m19.765976425s) [noise=true , address=77.90.31.10:4517 , piece_id=UH2TXG5FOJD7AMEWRP7GH2OLO7EU5VCWH7DMCNU2MRTLAP2VQTTQ, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.043614248 (59h52m19.766177279s) [noise=true , address=104.36.82.11:4115 , piece_id=4CYZEF64EJOYY3JBO7CRSCXAUQ6RDPODV2APJF6YK5BVZOOBFWVQ, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.043804493 (59h52m19.766367533s) [noise=true , address=104.36.82.16:4295 , piece_id=6DOLJJVJ4G6JYDGYB77A2MQ6ME3QBECFDJQ5UUGK3IPCGYPC74RQ, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:10.043987572 (59h52m19.766550602s) [noise=true , address=104.36.82.7:4911 , piece_id=3YCWZIU375IRKFLF7CJ23ILLGNACMUAJRGMBGCIHBG4AULJ6JC3A, seg_pos={0 0}, upload=57445]: Uploading piece... Jul 26 23:28:11.844014820 (59h52m21.56657786s) [upload=57445]: Next returned nil segment Jul 26 23:28:11.844031422 (59h52m21.566594462s) [upload=57445]: Waiting for error group managing segments... Jul 26 23:28:12.131862359 (59h52m21.854425399s) [noise=true , address=45.91.116.12:4225 , piece_id=JV7A7QI3XQQYIXD3RYC5Q5H7NRMIDSGQIK7O2B5UZL7I6JJ6HWMA, seg_pos={0 0}, upload=57445]: Done uploading piece. err: <nil> Jul 26 23:28:12.149430001 (59h52m21.871993041s) [noise=true , address=45.91.116.8:4045 , piece_id=DJCCE5JKJ434XETX4IXNPB4JZJ6CLGRTCJX2ZO6TGFDFKDDMJ6GQ, seg_pos={0 0}, upload=57445]: Done uploading piece. err: <nil> Jul 26 23:28:12.163538004 (59h52m21.886101034s) [noise=true , address=45.91.116.13:4154 , piece_id=FBQ2LUSOZTWNMYKQUUNYPZL4LFF3RNHKL3F3AIYDH7QVPMVR72JA, seg_pos={0 0}, upload=57445]: Done uploading piece. err: <nil> Jul 26 23:28:12.248874158 (59h52m21.971437198s) [noise=true , address=45.91.116.7:4009 , piece_id=FIXO7FBDDMTKTTHQTE77P5DHO4XAKIRQE5EWQDFJ4ZCXS2DBCOAQ, seg_pos={0 0}, upload=57445]: Done uploading piece. err: <nil> Jul 26 23:28:12.308828830 (59h52m22.03139187s) [noise=true , address=77.90.31.10:4517 , piece_id=UH2TXG5FOJD7AMEWRP7GH2OLO7EU5VCWH7DMCNU2MRTLAP2VQTTQ, seg_pos={0 0}, upload=57445]: Done uploading piece. err: <nil> Jul 26 23:28:39.165902245 (59h52m48.888465285s) [noise=true , address=104.36.82.9:4005 , piece_id=N66CUYLW5YR4GK6YCEFH72ZW66TDLHYJ6HIKZMLYZH7JSAC5LZZA, seg_pos={0 0}, upload=57445]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:34430->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:39.165946991 (59h52m48.888510021s) [noise=true , address=104.36.82.6:4874 , piece_id=2F2GVGVCGTF3LI6NB3NYFMPOLH43U4UCSOP3TNCYJ5UPAXT6OLAQ, seg_pos={0 0}, upload=57445]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:39458->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:39.165979082 (59h52m48.888542122s) [noise=true , address=104.36.82.11:4115 , piece_id=4CYZEF64EJOYY3JBO7CRSCXAUQ6RDPODV2APJF6YK5BVZOOBFWVQ, seg_pos={0 0}, upload=57445]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:56420->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:39.165999391 (59h52m48.888562431s) [noise=true , address=104.36.82.7:4911 , piece_id=3YCWZIU375IRKFLF7CJ23ILLGNACMUAJRGMBGCIHBG4AULJ6JC3A, seg_pos={0 0}, upload=57445]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:51822->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:39.166012876 (59h52m48.888575917s) [noise=true , address=104.36.82.2:4731 , piece_id=LFWL6CQKC6WV7XQ5M2JB7LIDMRTP4IAUDPDCBNA6POUWFR5LYGGA, seg_pos={0 0}, upload=57445]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:54032->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:39.166034829 (59h52m48.888597869s) [noise=true , address=104.36.82.15:4257 , piece_id=KI45UZJOTRLINZ77JN7ND5S64VUEFWLRKDZUE3CXHE6QNQQQFTSQ, seg_pos={0 0}, upload=57445]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:51850->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:39.166043675 (59h52m48.888606715s) [noise=true , address=104.36.82.4:4801 , piece_id=FMGQRE2T4FN63ILTVROVJL2YISPAAB6TJXFN2X2E6WBBADCHPX5Q, seg_pos={0 0}, upload=57445]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:53664->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:39.166065948 (59h52m48.888628978s) [noise=true , address=104.36.82.5:4839 , piece_id=EAHM5GORS2UK53HXLFJGGJPKE4IDRDTXRM5VU2CMB7OGTODAL5CQ, seg_pos={0 0}, upload=57445]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:56174->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:39.166079124 (59h52m48.888642164s) [noise=true , address=104.36.82.4:4804 , piece_id=LZG6ICH37TMXGQ4TFZT5POT7CV3LI7YSABTXQ25UTQJSH3T3PAUQ, seg_pos={0 0}, upload=57445]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:34274->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:39.166096286 (59h52m48.888659326s) [noise=true , address=104.36.82.8:4969 , piece_id=KCV3YOUGUXLR6BE2CLXWZRPUEATU6UJOPBJ5NGQNJZNMWNUK5Z3A, seg_pos={0 0}, upload=57445]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:36262->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:39.166110654 (59h52m48.888673694s) [noise=true , address=104.36.82.16:4295 , piece_id=6DOLJJVJ4G6JYDGYB77A2MQ6ME3QBECFDJQ5UUGK3IPCGYPC74RQ, seg_pos={0 0}, upload=57445]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:43442->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:39.166144228 (59h52m48.888707258s) [seg_pos={0 0}, upload=57445]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:28:39.392986896 (59h52m49.115549926s) [upload=57445]: Done uploading segments... Jul 26 23:28:39.393011884 (59h52m49.115574924s) [upload=57445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.228730364s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [..X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....X] duration: 1m0.000840225s FAIL (2025-07-26 23:28:17 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000878178s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.................................XXX.X..X.....XXXXX...XXX.X.] duration: 6.351927605s FAIL (2025-07-26 23:28:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.351949055s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.242633026s FAIL (2025-07-26 23:28:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 533s) [upload=57436]: Starting upload Jul 26 23:27:39.331257480 (59h51m49.05382052s) [upload=57436]: Done starting upload Jul 26 23:27:39.331339788 (59h51m49.053902818s) [upload=57436]: Uploading segments... Jul 26 23:27:39.331437205 (59h51m49.054000245s) [upload=57436]: Got next segment. Inline: false Jul 26 23:27:39.331484335 (59h51m49.054047435s) [upload=57436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:27:39.331494093 (59h51m49.054057153s) [upload=57436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:27:39.567084008 (59h51m49.289647049s) [seg_pos={0 0}, upload=57436]: Begin upload segment... Jul 26 23:27:39.567121981 (59h51m49.289685021s) [seg_pos={0 0}, upload=57436]: Done begin upload segment. Jul 26 23:27:39.567139374 (59h51m49.289702414s) [noise=true , address=104.36.82.4:4802 , piece_id=XQSFDXHNECC3RIGB7PKWBHVQG5MPXUXMHJDUD4CHGZ6DFRAIQBLA, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.567143512 (59h51m49.289706552s) [noise=true , address=104.36.82.2:4730 , piece_id=HXV3LQMNYXUHPKYMTDYX57OGWIJ7HIQZT6VVBBKMTCQELQ7WXF7A, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.567502860 (59h51m49.2900659s) [noise=true , address=45.91.116.6:4973 , piece_id=UUXIXKNM4VXWC7XELGDS22KHOWQVDLJ5RJ3W2T7KTV7DQPYJFXOQ, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.567767396 (59h51m49.290330436s) [noise=true , address=104.36.82.16:4295 , piece_id=3NHZFXVVBSO54ACD3GWHQCM654UMIA2Y423ZXOHKBGTWZ5PG3RAA, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.568028295 (59h51m49.290591335s) [noise=true , address=77.90.31.14:4661 , piece_id=YLWAN6TYROTJHTV37FPWU4AGV3D2H6E322RB7B2U6NKHKC27IVTQ, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.568370250 (59h51m49.29093329s) [noise=true , address=45.91.116.6:4974 , piece_id=TNVBSSSFN6BSU7LWTVCOLZDIXOYN33J57DL7IDJ64MARAUSEXDYQ, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.568611201 (59h51m49.291174231s) [noise=true , address=45.91.116.7:4010 , piece_id=UQYCMKVENAXBM45JPTEVIZ6JU6B3WWAYEQDRZGTVE7DOQQDSEVOQ, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.568860648 (59h51m49.291423688s) [noise=true , address=104.36.82.7:4910 , piece_id=5OV2RCQWAYE34NIMLQZRYESXDWPOHBQ7E6IYAZPJIP6K2EDAPZUA, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.569084707 (59h51m49.291647747s) [noise=true , address=104.36.82.6:4875 , piece_id=MQP6UGMAXUEW5KIEUQCDMXYRUGQPAWLQ5XAGJKEVZR4PSEHGWS3A, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.569293055 (59h51m49.291856095s) [noise=true , address=45.91.116.10:4083 , piece_id=FCRSB7OYDFDCCHMYA4NLQ5VLQ2X7YJ775CGQWDEGGQOBPSA3O3FQ, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.569487558 (59h51m49.292050598s) [noise=true , address=104.36.82.15:4258 , piece_id=NJRQFVHRMB4RRD7UT6JSVFD7RUIRNVB7ZMOOG52L6NN6YTXZBN6Q, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.569693301 (59h51m49.292256351s) [noise=true , address=77.90.31.21:4877 , piece_id=QHU45QJ3X6HG6KI4OO2RMTC5ZTLLGZZPPGU424HEKZYLQBFCWEGA, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.569938119 (59h51m49.292501159s) [noise=true , address=45.91.116.14:4190 , piece_id=DNN3P6KT4T7EJ4KOEA3P4CPZHYUMVV2ZTCSP66DBZJH2I6P4OC4Q, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.570274533 (59h51m49.292837573s) [noise=true , address=104.36.82.9:4007 , piece_id=UIWLOCHJJSMUMJUHVP4PA4MYGW5CE43VOR65B4RPP4EVZ4YMXHOQ, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.570485116 (59h51m49.293048156s) [noise=true , address=45.91.116.12:4225 , piece_id=RGRDAVTPQXB6BHFQLPCJTXRU37X3EDENWE7RL2I53WXNSNRS4WEQ, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:39.570688515 (59h51m49.293251555s) [noise=true , address=104.36.82.5:4839 , piece_id=JMY6EIFM7IDUV7UOVP6JG2WHNUBKGHIBOZEG4I57LQ4QJCQW763A, seg_pos={0 0}, upload=57436]: Uploading piece... Jul 26 23:27:41.293264403 (59h51m51.015827454s) [upload=57436]: Next returned nil segment Jul 26 23:27:41.293277017 (59h51m51.015840057s) [upload=57436]: Waiting for error group managing segments... Jul 26 23:27:41.605319228 (59h51m51.327882268s) [noise=true , address=45.91.116.14:4190 , piece_id=DNN3P6KT4T7EJ4KOEA3P4CPZHYUMVV2ZTCSP66DBZJH2I6P4OC4Q, seg_pos={0 0}, upload=57436]: Done uploading piece. err: <nil> Jul 26 23:27:41.608520413 (59h51m51.331083453s) [noise=true , address=77.90.31.14:4661 , piece_id=YLWAN6TYROTJHTV37FPWU4AGV3D2H6E322RB7B2U6NKHKC27IVTQ, seg_pos={0 0}, upload=57436]: Done uploading piece. err: <nil> Jul 26 23:27:41.668572731 (59h51m51.391135772s) [noise=true , address=45.91.116.6:4973 , piece_id=UUXIXKNM4VXWC7XELGDS22KHOWQVDLJ5RJ3W2T7KTV7DQPYJFXOQ, seg_pos={0 0}, upload=57436]: Done uploading piece. err: <nil> Jul 26 23:27:41.677265132 (59h51m51.399828173s) [noise=true , address=45.91.116.10:4083 , piece_id=FCRSB7OYDFDCCHMYA4NLQ5VLQ2X7YJ775CGQWDEGGQOBPSA3O3FQ, seg_pos={0 0}, upload=57436]: Done uploading piece. err: <nil> Jul 26 23:27:41.681618942 (59h51m51.404181982s) [noise=true , address=45.91.116.6:4974 , piece_id=TNVBSSSFN6BSU7LWTVCOLZDIXOYN33J57DL7IDJ64MARAUSEXDYQ, seg_pos={0 0}, upload=57436]: Done uploading piece. err: <nil> Jul 26 23:27:41.692558523 (59h51m51.415121563s) [noise=true , address=77.90.31.21:4877 , piece_id=QHU45QJ3X6HG6KI4OO2RMTC5ZTLLGZZPPGU424HEKZYLQBFCWEGA, seg_pos={0 0}, upload=57436]: Done uploading piece. err: <nil> Jul 26 23:27:41.756668415 (59h51m51.479231455s) [noise=true , address=45.91.116.7:4010 , piece_id=UQYCMKVENAXBM45JPTEVIZ6JU6B3WWAYEQDRZGTVE7DOQQDSEVOQ, seg_pos={0 0}, upload=57436]: Done uploading piece. err: <nil> Jul 26 23:27:41.833080425 (59h51m51.555643465s) [noise=true , address=45.91.116.12:4225 , piece_id=RGRDAVTPQXB6BHFQLPCJTXRU37X3EDENWE7RL2I53WXNSNRS4WEQ, seg_pos={0 0}, upload=57436]: Done uploading piece. err: <nil> Jul 26 23:28:08.681130988 (59h52m18.403694028s) [noise=true , address=104.36.82.2:4730 , piece_id=HXV3LQMNYXUHPKYMTDYX57OGWIJ7HIQZT6VVBBKMTCQELQ7WXF7A, seg_pos={0 0}, upload=57436]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:55514->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:08.681174461 (59h52m18.403737501s) [noise=true , address=104.36.82.4:4802 , piece_id=XQSFDXHNECC3RIGB7PKWBHVQG5MPXUXMHJDUD4CHGZ6DFRAIQBLA, seg_pos={0 0}, upload=57436]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:08.681205009 (59h52m18.403768049s) [noise=true , address=104.36.82.16:4295 , piece_id=3NHZFXVVBSO54ACD3GWHQCM654UMIA2Y423ZXOHKBGTWZ5PG3RAA, seg_pos={0 0}, upload=57436]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:08.681239436 (59h52m18.403802466s) [noise=true , address=104.36.82.9:4007 , piece_id=UIWLOCHJJSMUMJUHVP4PA4MYGW5CE43VOR65B4RPP4EVZ4YMXHOQ, seg_pos={0 0}, upload=57436]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:47174->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:08.681266055 (59h52m18.403829096s) [noise=true , address=104.36.82.5:4839 , piece_id=JMY6EIFM7IDUV7UOVP6JG2WHNUBKGHIBOZEG4I57LQ4QJCQW763A, seg_pos={0 0}, upload=57436]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:08.681275414 (59h52m18.403838454s) [noise=true , address=104.36.82.7:4910 , piece_id=5OV2RCQWAYE34NIMLQZRYESXDWPOHBQ7E6IYAZPJIP6K2EDAPZUA, seg_pos={0 0}, upload=57436]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:08.681296104 (59h52m18.403859144s) [noise=true , address=104.36.82.6:4875 , piece_id=MQP6UGMAXUEW5KIEUQCDMXYRUGQPAWLQ5XAGJKEVZR4PSEHGWS3A, seg_pos={0 0}, upload=57436]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:08.681319578 (59h52m18.403882618s) [noise=true , address=104.36.82.15:4258 , piece_id=NJRQFVHRMB4RRD7UT6JSVFD7RUIRNVB7ZMOOG52L6NN6YTXZBN6Q, seg_pos={0 0}, upload=57436]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:42142->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:08.681343354 (59h52m18.403906394s) [seg_pos={0 0}, upload=57436]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:28:08.922251474 (59h52m18.644814514s) [upload=57436]: Done uploading segments... Jul 26 23:28:08.922275359 (59h52m18.644838399s) [upload=57436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.242653805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.907652455s FAIL (2025-07-26 23:28:05 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.907675309s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23096054s FAIL (2025-07-26 23:27:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== IA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.294275546 (59h51m2.016838576s) [noise=true , address=104.36.82.22:4510 , piece_id=CVVXR3BL3V2QGZC2M3CKDMDJWWNYHQ5GZ4GI2EMYTQQXZMGGEPDA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.294523941 (59h51m2.017086981s) [noise=true , address=104.36.82.53:4127 , piece_id=Y7AOQ6E64QQLOVIO4HQRWAJRMOFRFPRTYPTO4O3PBYNC2ZGNHZJQ, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.295386402 (59h51m2.017949442s) [noise=true , address=104.36.82.46:4490 , piece_id=F5SINJ2OCMIH7SMEG5IOKPLGNQDGOJ3JWVBEXQ5ZV4N2FILG5R7A, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.295669744 (59h51m2.018232784s) [noise=true , address=79.127.219.47:4048 , piece_id=ZNYUAXIUKUQRWQT536X2PEKHAORKGUNUWTSDGMP3HPAOS2LGYBLQ, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.295942115 (59h51m2.018505155s) [noise=true , address=156.146.43.225:4453 , piece_id=7FM3YH327FRUDPLPJM35UKUJMFQ3UPP56OSXS7MNW6B3GKBRXMWA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.296407606 (59h51m2.018970636s) [noise=true , address=104.36.82.21:4503 , piece_id=JROIOE56GPG2AJT7HWSZUMMZFWJ7NFB6LPB3Z3JJZBO5CC2AC3IQ, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.296637325 (59h51m2.019200365s) [noise=true , address=104.36.82.52:4097 , piece_id=J5VOVTTNSU2XPBTCEAPAAJOFTPM364R7CCHRVC52PAJGSKEJNH3Q, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.296832348 (59h51m2.019395388s) [noise=true , address=79.127.205.246:4272 , piece_id=5B7LE4Z25OUGZGH7WQBF5KV2G2UP3QKAB5TTUUKNY2FEAQ62KTOQ, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.297034936 (59h51m2.019597976s) [noise=true , address=79.127.205.235:4682 , piece_id=SV5NQJWIUQMAMSO264FCRZ6N7BD2GJXDSXYURGLU23WOBY2TUK3A, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.297261900 (59h51m2.01982494s) [noise=true , address=79.127.205.234:4619 , piece_id=3BPZ3VHTHETHWDC4XIE42QFAUKAS47T2MLYRZ3XGJ5JLTAYFXLAA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.297468967 (59h51m2.020032007s) [noise=true , address=104.36.82.54:4157 , piece_id=HJM5A37YBHG4O2N3W6KWBNFZWHDHOXFZ7YI3FLUXGWNV3ALX723Q, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.297713734 (59h51m2.020276774s) [noise=true , address=79.127.205.245:4221 , piece_id=QW4VM7F7MDOIDFQT32R5REAPLI7ZZTS4HZNFG3BEGX4BTHN6XOOQ, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.298040980 (59h51m2.02060402s) [noise=true , address=104.36.82.47:4523 , piece_id=UDBVQ2UII4ZJGW6HPMXG4UMPXW46KVN7YHFIQ5UUXGPUNQOK6O7Q, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.298270851 (59h51m2.020833891s) [noise=true , address=79.127.220.101:4048 , piece_id=EY4DOTTE72JJMXUWA7MVQ2Z6ENRVPLSMVGCXRON7FMI5FDQV5TBA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.298378517 (59h51m2.020941557s) [noise=true , address=104.36.82.45:4924 , piece_id=LFONCFVCAKPEQ7PWAU5PXJ4C7FRFENACAV352WEMWRYYDETBUACA, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:52.298636640 (59h51m2.02119968s) [noise=true , address=79.127.219.38:4017 , piece_id=J66T5WVTOZV7MDOEE7ICJNQNPCP3EXRZWENUPQWRNR5S7RE5QK5Q, seg_pos={0 0}, upload=57424]: Uploading piece... Jul 26 23:26:53.383003392 (59h51m3.105566432s) [upload=57424]: Next returned nil segment Jul 26 23:26:53.383020544 (59h51m3.105583585s) [upload=57424]: Waiting for error group managing segments... Jul 26 23:26:53.686724482 (59h51m3.409287512s) [noise=true , address=79.127.205.235:4682 , piece_id=SV5NQJWIUQMAMSO264FCRZ6N7BD2GJXDSXYURGLU23WOBY2TUK3A, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:53.699720738 (59h51m3.422283778s) [noise=true , address=79.127.205.234:4619 , piece_id=3BPZ3VHTHETHWDC4XIE42QFAUKAS47T2MLYRZ3XGJ5JLTAYFXLAA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:53.762019154 (59h51m3.484582184s) [noise=true , address=79.127.205.246:4272 , piece_id=5B7LE4Z25OUGZGH7WQBF5KV2G2UP3QKAB5TTUUKNY2FEAQ62KTOQ, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:53.939190366 (59h51m3.661753406s) [noise=true , address=79.127.205.245:4221 , piece_id=QW4VM7F7MDOIDFQT32R5REAPLI7ZZTS4HZNFG3BEGX4BTHN6XOOQ, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:54.217973492 (59h51m3.940536532s) [noise=true , address=77.90.31.17:4796 , piece_id=VJPFUZI7545BTAW5OU2K6HAK6D2CJKJW6AMOA7LNPY5SMVVKBN2A, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:54.339267404 (59h51m4.061830444s) [noise=true , address=156.146.43.225:4453 , piece_id=7FM3YH327FRUDPLPJM35UKUJMFQ3UPP56OSXS7MNW6B3GKBRXMWA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:54.462463476 (59h51m4.185026516s) [noise=true , address=79.127.219.38:4017 , piece_id=J66T5WVTOZV7MDOEE7ICJNQNPCP3EXRZWENUPQWRNR5S7RE5QK5Q, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:57.286145912 (59h51m7.008708952s) [noise=true , address=79.127.219.47:4048 , piece_id=ZNYUAXIUKUQRWQT536X2PEKHAORKGUNUWTSDGMP3HPAOS2LGYBLQ, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:26:59.466725590 (59h51m9.18928863s) [noise=true , address=79.127.220.101:4048 , piece_id=EY4DOTTE72JJMXUWA7MVQ2Z6ENRVPLSMVGCXRON7FMI5FDQV5TBA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: <nil> Jul 26 23:27:51.196008723 (59h52m0.918571753s) [noise=true , address=104.36.82.52:4097 , piece_id=J5VOVTTNSU2XPBTCEAPAAJOFTPM364R7CCHRVC52PAJGSKEJNH3Q, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM, address:104.36.82.52:4097): protocol: write tcp 188.166.225.170:53868->104.36.82.52:4097: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196055173 (59h52m0.918618213s) [noise=true , address=104.36.82.22:4510 , piece_id=CVVXR3BL3V2QGZC2M3CKDMDJWWNYHQ5GZ4GI2EMYTQQXZMGGEPDA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:104.36.82.22:4510): protocol: write tcp 188.166.225.170:34290->104.36.82.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196089328 (59h52m0.918652368s) [noise=true , address=104.36.82.47:4523 , piece_id=UDBVQ2UII4ZJGW6HPMXG4UMPXW46KVN7YHFIQ5UUXGPUNQOK6O7Q, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: write tcp 188.166.225.170:43514->104.36.82.47:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196258872 (59h52m0.918821912s) [noise=true , address=104.36.82.45:4924 , piece_id=LFONCFVCAKPEQ7PWAU5PXJ4C7FRFENACAV352WEMWRYYDETBUACA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:104.36.82.45:4924): protocol: write tcp 188.166.225.170:53286->104.36.82.45:4924: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196295432 (59h52m0.918858472s) [noise=true , address=104.36.82.53:4127 , piece_id=Y7AOQ6E64QQLOVIO4HQRWAJRMOFRFPRTYPTO4O3PBYNC2ZGNHZJQ, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:12VM12t6Dq6SyApviDYSSUqWnnzW3wTfA4o8VsvJaah4fLj9EWh, address:104.36.82.53:4127): protocol: write tcp 188.166.225.170:46992->104.36.82.53:4127: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196352300 (59h52m0.91891534s) [noise=true , address=104.36.82.54:4157 , piece_id=HJM5A37YBHG4O2N3W6KWBNFZWHDHOXFZ7YI3FLUXGWNV3ALX723Q, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: rpc: dial tcp 104.36.82.54:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196377118 (59h52m0.918940158s) [noise=true , address=104.36.82.46:4490 , piece_id=F5SINJ2OCMIH7SMEG5IOKPLGNQDGOJ3JWVBEXQ5ZV4N2FILG5R7A, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: write tcp 188.166.225.170:59674->104.36.82.46:4490: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:51.196431993 (59h52m0.918995033s) [noise=true , address=104.36.82.56:4237 , piece_id=TZOUNQCHWEOCGPJUIT3NAX7HLYSMVLHXHQIH3T37FTUQF25YLYIA, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:1VzewCRbNpJv7kNwp9wruQuoVSaZFjA44qN33Ct7qm1GnfhtDj, address:104.36.82.56:4237): protocol: write tcp 188.166.225.170:45760->104.36.82.56:4237: use of closed network connection; conn closed: op Jul 26 23:27:51.196464665 (59h52m0.919027705s) [noise=true , address=104.36.82.21:4503 , piece_id=JROIOE56GPG2AJT7HWSZUMMZFWJ7NFB6LPB3Z3JJZBO5CC2AC3IQ, seg_pos={0 0}, upload=57424]: Done uploading piece. err: ecclient: upload failed (node:17F3Gn5LYgekp8WgagEdFXtRu8jCc27qA9UuW7pWsJ9FUFKFSi, address:104.36.82.21:4503): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:51.196483441 (59h52m0.919046481s) [seg_pos={0 0}, upload=57424]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:27:51.425876329 (59h52m1.148439369s) [upload=57424]: Done uploading segments... Jul 26 23:27:51.425900656 (59h52m1.148463696s) [upload=57424]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.230972543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..................................XXX.X..X.....XXXXX...XXX.X] duration: 30.001517868s FAIL (2025-07-26 23:27:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001736485s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.878870718s FAIL (2025-07-26 23:27:41 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.878901335s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [....X.X...........................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.246061562s FAIL (2025-07-26 23:27:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.246082703s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [...X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....] duration: 58.700563905s FAIL (2025-07-26 23:27:14 UTC) orbiter warning: context deadline exceeded Duration: 58.700577069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..X.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235588785s FAIL (2025-07-26 23:27:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== pP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:48030->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.254459354 (59h50m37.977022394s) [seg_pos={0 0}, upload=57410]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:26:28.483531338 (59h50m38.206094378s) [upload=57410]: Done uploading segments... Jul 26 23:26:28.483565944 (59h50m38.206128984s) [upload=57410]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:26:38.771605288 (59h50m48.494168318s) [upload=57420]: Starting upload Jul 26 23:26:38.771663028 (59h50m48.494226068s) [upload=57420]: Done starting upload Jul 26 23:26:38.771740014 (59h50m48.494303055s) [upload=57420]: Uploading segments... Jul 26 23:26:38.771873771 (59h50m48.494436811s) [upload=57420]: Got next segment. Inline: false Jul 26 23:26:38.771913908 (59h50m48.494476938s) [upload=57420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:26:38.771918927 (59h50m48.494481967s) [upload=57420]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:26:39.006837118 (59h50m48.729400158s) [seg_pos={0 0}, upload=57420]: Begin upload segment... Jul 26 23:26:39.006891361 (59h50m48.729454401s) [seg_pos={0 0}, upload=57420]: Done begin upload segment. Jul 26 23:26:39.006912592 (59h50m48.729475622s) [noise=true , address=104.36.82.9:4007 , piece_id=P7TIRD2A5HRONEHGIYKI5ULPPOIUUYVV4I5IFPQUBLJWI75NUGSA, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.006963258 (59h50m48.729526308s) [noise=true , address=45.91.116.11:4118 , piece_id=2T74KVVZVO3Q5AHCZJ67US6YZZDCS4NZTZD4K3OZMHIUYIX2NTQA, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.007285645 (59h50m48.729848736s) [noise=true , address=45.91.116.10:4081 , piece_id=Y5PEWGD7B2SIWPRRYXXMVE6MV4EQGP3AIVS3RFZ32XU3UR3NEGUA, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.007630836 (59h50m48.730193916s) [noise=true , address=45.91.116.6:4975 , piece_id=D5BQS7TOYDLZFHWDG3SDL6CN2QGTBMAYHFGTI4JM4YTDOXTDQVBQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.007647027 (59h50m48.730210057s) [noise=true , address=45.91.116.14:4190 , piece_id=XD3MUL3RPM6POZL2JYASE3PZ6R4COMAONW6NLMQXRZGMUL3B2M7A, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.007980264 (59h50m48.730543304s) [noise=true , address=104.36.82.11:4114 , piece_id=KCFR4KAQTOWXIRPKUJTBOGM4U6QI6L5G5KJGV463DSU7ZHUA2YXQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.008315066 (59h50m48.730878096s) [noise=true , address=77.90.31.15:4698 , piece_id=ZAKGOZWCXTURYSWUNL7BNINS5FX5OGIBQGT6HFCFEVPB5NGQODLQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.008757863 (59h50m48.731320903s) [noise=true , address=45.91.116.4:4940 , piece_id=QTVD733LQWLVOI4NU3LCSFTKXJ7QPA66TTNJJPEFTKUJSIORR3HQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.008975409 (59h50m48.731538449s) [noise=true , address=45.91.116.13:4155 , piece_id=XMGAUAQUV433OEFZEUJE4TTTB3XLJKMWFWR3DQ7L2S2IEKAQRYKQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.009195079 (59h50m48.731758119s) [noise=true , address=45.91.116.7:4010 , piece_id=EGLJ4I3WHMDKSAPN5OHMORWLNOXUNF7M3ZF5HQMPWTN7ET4JAZUQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.009398478 (59h50m48.731961518s) [noise=true , address=104.36.82.5:4839 , piece_id=2QH4ZYQAAO76IJ4NAKR6KJCDAC4X2FIA4XUIVWGZYFYMEBXKB74A, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.009607819 (59h50m48.732170849s) [noise=true , address=45.91.116.12:4227 , piece_id=OY6TMBDTHHOSER5MEVREOPZK6ZO4HU5SMVSNDR6Q2GO66IPTMCKQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.009930716 (59h50m48.732493756s) [noise=true , address=104.36.82.4:4803 , piece_id=UGKRKHX3A6PBLOY65L4GQF2BKOQO3VIKCJRQHINIEYN5LOG3NU7Q, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.010137292 (59h50m48.732700332s) [noise=true , address=104.36.82.3:4767 , piece_id=U22E5JPNZ7QT57KRX22KXYXAWX2QFMRTNWE6KR4EOUYZMMPIASUQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.010322747 (59h50m48.732885777s) [noise=true , address=45.91.116.8:4047 , piece_id=HI433WOL4GHJMYSGA5CCFFSXCQIXTDM27S2X5DQUGR3VU77XHIKA, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:39.010513903 (59h50m48.733076933s) [noise=true , address=104.36.82.8:4969 , piece_id=QXI4TD4DKECLPMTCB3AVKX2ZQT4UBX26LZEIX5YS7PHFMFU65WWQ, seg_pos={0 0}, upload=57420]: Uploading piece... Jul 26 23:26:40.734893128 (59h50m50.457456168s) [upload=57420]: Next returned nil segment Jul 26 23:26:40.734910742 (59h50m50.457473782s) [upload=57420]: Waiting for error group managing segments... Jul 26 23:26:41.011293566 (59h50m50.733856606s) [noise=true , address=45.91.116.7:4010 , piece_id=EGLJ4I3WHMDKSAPN5OHMORWLNOXUNF7M3ZF5HQMPWTN7ET4JAZUQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.042708983 (59h50m50.765272023s) [noise=true , address=45.91.116.12:4227 , piece_id=OY6TMBDTHHOSER5MEVREOPZK6ZO4HU5SMVSNDR6Q2GO66IPTMCKQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.101344720 (59h50m50.823907761s) [noise=true , address=45.91.116.13:4155 , piece_id=XMGAUAQUV433OEFZEUJE4TTTB3XLJKMWFWR3DQ7L2S2IEKAQRYKQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.110380439 (59h50m50.832943499s) [noise=true , address=45.91.116.4:4940 , piece_id=QTVD733LQWLVOI4NU3LCSFTKXJ7QPA66TTNJJPEFTKUJSIORR3HQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.117188665 (59h50m50.839751706s) [noise=true , address=45.91.116.8:4047 , piece_id=HI433WOL4GHJMYSGA5CCFFSXCQIXTDM27S2X5DQUGR3VU77XHIKA, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.159493197 (59h50m50.882056237s) [noise=true , address=45.91.116.11:4118 , piece_id=2T74KVVZVO3Q5AHCZJ67US6YZZDCS4NZTZD4K3OZMHIUYIX2NTQA, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.191547065 (59h50m50.914110105s) [noise=true , address=45.91.116.10:4081 , piece_id=Y5PEWGD7B2SIWPRRYXXMVE6MV4EQGP3AIVS3RFZ32XU3UR3NEGUA, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.268746192 (59h50m50.991309272s) [noise=true , address=45.91.116.6:4975 , piece_id=D5BQS7TOYDLZFHWDG3SDL6CN2QGTBMAYHFGTI4JM4YTDOXTDQVBQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:26:41.587100358 (59h50m51.309663399s) [noise=true , address=45.91.116.14:4190 , piece_id=XD3MUL3RPM6POZL2JYASE3PZ6R4COMAONW6NLMQXRZGMUL3B2M7A, seg_pos={0 0}, upload=57420]: Done uploading piece. err: <nil> Jul 26 23:27:08.124456439 (59h51m17.847019469s) [noise=true , address=77.90.31.15:4698 , piece_id=ZAKGOZWCXTURYSWUNL7BNINS5FX5OGIBQGT6HFCFEVPB5NGQODLQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:27:08.124500803 (59h51m17.847063843s) [noise=true , address=104.36.82.4:4803 , piece_id=UGKRKHX3A6PBLOY65L4GQF2BKOQO3VIKCJRQHINIEYN5LOG3NU7Q, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 188.166.225.170:40276->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:08.124593471 (59h51m17.847156511s) [noise=true , address=104.36.82.9:4007 , piece_id=P7TIRD2A5HRONEHGIYKI5ULPPOIUUYVV4I5IFPQUBLJWI75NUGSA, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:08.124637275 (59h51m17.847200305s) [noise=true , address=104.36.82.8:4969 , piece_id=QXI4TD4DKECLPMTCB3AVKX2ZQT4UBX26LZEIX5YS7PHFMFU65WWQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:08.124652394 (59h51m17.847215434s) [noise=true , address=104.36.82.5:4839 , piece_id=2QH4ZYQAAO76IJ4NAKR6KJCDAC4X2FIA4XUIVWGZYFYMEBXKB74A, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:08.124700917 (59h51m17.847263957s) [noise=true , address=104.36.82.3:4767 , piece_id=U22E5JPNZ7QT57KRX22KXYXAWX2QFMRTNWE6KR4EOUYZMMPIASUQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:08.124841726 (59h51m17.847404766s) [noise=true , address=104.36.82.11:4114 , piece_id=KCFR4KAQTOWXIRPKUJTBOGM4U6QI6L5G5KJGV463DSU7ZHUA2YXQ, seg_pos={0 0}, upload=57420]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; context deadline exceeded Jul 26 23:27:08.124867395 (59h51m17.847430436s) [seg_pos={0 0}, upload=57420]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:27:08.358243524 (59h51m18.080806564s) [upload=57420]: Done uploading segments... Jul 26 23:27:08.358267780 (59h51m18.08083082s) [upload=57420]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.235614014s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.836120603s FAIL (2025-07-26 23:26:56 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.836147334s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [.....X.X...........................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.253141912s FAIL (2025-07-26 23:26:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.253163302s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230726148s FAIL (2025-07-26 23:26:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== piece_id=FQMQIUHWZLKSOENOPOIN7ZVRCPJHB3SHPSNHMYDUKTXCTESH6R3Q, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.821553899 (59h50m1.544116939s) [noise=true , address=104.36.82.47:4526 , piece_id=IIGEOKA4DAH4IRVTOGTJL7D7NITJ3QJEEY7UDHMMRVN4MOU366NA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.821899880 (59h50m1.54446292s) [noise=true , address=79.127.219.51:4212 , piece_id=7MIJBNKQIVZ6T5EXJTO3OZYH5PHIZVKLVMO57Y6DHPENDMPKM7ZQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.822146412 (59h50m1.544709442s) [noise=true , address=77.90.31.19:4844 , piece_id=G5SLJFSN7JSJ2HRHGSAFVOJ7YZXZLP2BVKPU4U454SCTU4QSNW5A, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.822380250 (59h50m1.54494329s) [noise=true , address=77.90.31.16:4764 , piece_id=4USHBAIOC73YCCFP4YEF3Q4GSME3W4IKXNFIJMWD2AKGHKVZM4TQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.822668661 (59h50m1.545231701s) [noise=true , address=77.90.31.21:4896 , piece_id=PLNV77UGJCLZIRPDAHD4CJGHREQCDAM4KCLUVAV3JVF4C4YL6VKQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.822916886 (59h50m1.545479926s) [noise=true , address=156.146.43.225:4448 , piece_id=ERCFAATUEH2XE7ALBSGTOEIPFH25TM53EJHEWYLV4BEHUTKDMV4A, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.823099245 (59h50m1.545662275s) [noise=true , address=64.49.13.138:4483 , piece_id=2IQ4426FJSTRFFLZDFISRORQYNJABGIPNCCSOHVB2INYSV4UAZRQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.826943901 (59h50m1.549506932s) [noise=true , address=104.36.82.53:4145 , piece_id=PBSW3KGSECAZU5TL7MRDU5CLXHA4RNG3VG3UX4FSQ2VTXF5CVENA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.827449258 (59h50m1.550012298s) [noise=true , address=64.49.13.138:4482 , piece_id=WBXKKSURYBGJ6PAUYI6YMZJKTAWEFE4V34CXDPZLFTLD3BNMDSSQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.828169065 (59h50m1.550732105s) [noise=true , address=79.127.205.246:4281 , piece_id=MA6GFDKRJJTUO35ZVN6ZSFAFCMYKMEPRGSIT6WLABNYW324AO4MQ, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.828891117 (59h50m1.551454157s) [noise=true , address=79.127.205.234:4604 , piece_id=EUGQO2NAE5S47TAA6Y77H5NUDQDIX54GNHR7RU6TFY4ASA47GYQA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.829584223 (59h50m1.552147263s) [noise=true , address=104.36.82.54:4182 , piece_id=QL6CL4GCFPHADND446NNNR2ZE74D427OGFOWZTHTTUPCQMINIC6Q, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.830273041 (59h50m1.552836081s) [noise=true , address=104.36.82.52:4109 , piece_id=YWDKYIVYVYERBSADC7THF27SDSZVQRZE67IUJ4UUKTW7TWBQPJRA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.830674710 (59h50m1.55323775s) [noise=true , address=104.36.82.46:4481 , piece_id=CPXE7BBSEZH7PLIEVSWCUVUOJS2PQVKMTUMMHSQD5SCQICAGDG3A, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.831361854 (59h50m1.553924894s) [noise=true , address=79.127.205.245:4239 , piece_id=QA3GG5AXGIYCVKVXDGW5WJXPNWGGTFEPP3GH6FMIC3LLWC2SBUWA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:51.831652079 (59h50m1.554215109s) [noise=true , address=77.90.31.15:4722 , piece_id=Y4G5PT5SWPMLMSAFG2DQ2RLNXZ5RHFDZKP5WRA666MHPXAPURFJA, seg_pos={0 0}, upload=57408]: Uploading piece... Jul 26 23:25:52.842054214 (59h50m2.564617244s) [noise=true , address=156.146.43.225:4448 , piece_id=ERCFAATUEH2XE7ALBSGTOEIPFH25TM53EJHEWYLV4BEHUTKDMV4A, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 23:25:52.928749428 (59h50m2.651312468s) [upload=57408]: Next returned nil segment Jul 26 23:25:52.928760620 (59h50m2.65132366s) [upload=57408]: Waiting for error group managing segments... Jul 26 23:25:53.207960262 (59h50m2.930523302s) [noise=true , address=79.127.205.245:4239 , piece_id=QA3GG5AXGIYCVKVXDGW5WJXPNWGGTFEPP3GH6FMIC3LLWC2SBUWA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:53.217235309 (59h50m2.939798349s) [noise=true , address=64.49.13.138:4482 , piece_id=WBXKKSURYBGJ6PAUYI6YMZJKTAWEFE4V34CXDPZLFTLD3BNMDSSQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:25:53.229690869 (59h50m2.952253909s) [noise=true , address=79.127.205.246:4281 , piece_id=MA6GFDKRJJTUO35ZVN6ZSFAFCMYKMEPRGSIT6WLABNYW324AO4MQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:53.261788051 (59h50m2.984351091s) [noise=true , address=79.127.205.234:4604 , piece_id=EUGQO2NAE5S47TAA6Y77H5NUDQDIX54GNHR7RU6TFY4ASA47GYQA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:53.729234437 (59h50m3.451797477s) [noise=true , address=45.91.116.9:4277 , piece_id=QHZHUJ5VTG3QTK4UYJNJIQRCKBCTPTEWIMU6W4U4YFI3AGGIE4EA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:53.796795362 (59h50m3.519358402s) [noise=true , address=79.127.219.38:4003 , piece_id=FQMQIUHWZLKSOENOPOIN7ZVRCPJHB3SHPSNHMYDUKTXCTESH6R3Q, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:53.936662861 (59h50m3.659225901s) [noise=true , address=64.49.13.138:4483 , piece_id=2IQ4426FJSTRFFLZDFISRORQYNJABGIPNCCSOHVB2INYSV4UAZRQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:25:56.306140127 (59h50m6.028703157s) [noise=true , address=79.127.219.51:4212 , piece_id=7MIJBNKQIVZ6T5EXJTO3OZYH5PHIZVKLVMO57Y6DHPENDMPKM7ZQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:26:23.113950228 (59h50m32.836513258s) [noise=true , address=77.90.31.19:4844 , piece_id=G5SLJFSN7JSJ2HRHGSAFVOJ7YZXZLP2BVKPU4U454SCTU4QSNW5A, seg_pos={0 0}, upload=57408]: Done uploading piece. err: <nil> Jul 26 23:26:50.711037243 (59h51m0.433600283s) [noise=true , address=77.90.31.15:4722 , piece_id=Y4G5PT5SWPMLMSAFG2DQ2RLNXZ5RHFDZKP5WRA666MHPXAPURFJA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:26:50.711246825 (59h51m0.433809875s) [noise=true , address=104.36.82.47:4526 , piece_id=IIGEOKA4DAH4IRVTOGTJL7D7NITJ3QJEEY7UDHMMRVN4MOU366NA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: rpc: dial tcp 104.36.82.47:4526: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711307010 (59h51m0.43387006s) [noise=true , address=77.90.31.16:4764 , piece_id=4USHBAIOC73YCCFP4YEF3Q4GSME3W4IKXNFIJMWD2AKGHKVZM4TQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, address:77.90.31.16:4764): protocol: rpc: dial tcp 77.90.31.16:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711378777 (59h51m0.433941827s) [noise=true , address=77.90.31.21:4896 , piece_id=PLNV77UGJCLZIRPDAHD4CJGHREQCDAM4KCLUVAV3JVF4C4YL6VKQ, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711442028 (59h51m0.434005088s) [noise=true , address=104.36.82.46:4481 , piece_id=CPXE7BBSEZH7PLIEVSWCUVUOJS2PQVKMTUMMHSQD5SCQICAGDG3A, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: write tcp 188.166.225.170:39674->104.36.82.46:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711496893 (59h51m0.434059933s) [noise=true , address=104.36.82.54:4182 , piece_id=QL6CL4GCFPHADND446NNNR2ZE74D427OGFOWZTHTTUPCQMINIC6Q, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:12HbkH31sfDUnZLXAbmzQphcFYmzB8MQo98Q6Ufbo59et42zxMy, address:104.36.82.54:4182): protocol: rpc: dial tcp 104.36.82.54:4182: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711539404 (59h51m0.434102444s) [noise=true , address=104.36.82.53:4145 , piece_id=PBSW3KGSECAZU5TL7MRDU5CLXHA4RNG3VG3UX4FSQ2VTXF5CVENA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: write tcp 188.166.225.170:44082->104.36.82.53:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711619016 (59h51m0.434182046s) [noise=true , address=104.36.82.52:4109 , piece_id=YWDKYIVYVYERBSADC7THF27SDSZVQRZE67IUJ4UUKTW7TWBQPJRA, seg_pos={0 0}, upload=57408]: Done uploading piece. err: ecclient: upload failed (node:1hBqAk4HrRmUKRw3RS7xUHZ4rdUMiF25jKWF7jqNrffZG8pEcj, address:104.36.82.52:4109): protocol: rpc: dial tcp 104.36.82.52:4109: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:50.711637302 (59h51m0.434200342s) [seg_pos={0 0}, upload=57408]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:26:50.939918903 (59h51m0.662481944s) [upload=57408]: Done uploading segments... Jul 26 23:26:50.939941897 (59h51m0.662504937s) [upload=57408]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.230738451s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)