Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X..X.X..............X......................................] duration: 5.534127223s FAIL (2025-07-27 01:22:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.534153836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XXXX.....X....X....XXX.....................................] duration: 5.936690236s FAIL (2025-07-27 01:22:19 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.936719844s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXX..X.X.X..........X....................................] duration: 5.734573667s FAIL (2025-07-27 01:22:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.734600347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...................X....X..........X...................X....] duration: 1.851812749s FAIL (2025-07-27 01:22:02 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 1.851845642s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X..X.X..............X.....................................] duration: 5.635017659s FAIL (2025-07-27 01:21:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.635043798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....................X....X..........X...................X...] duration: 2.032757661s FAIL (2025-07-27 01:21:34 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 2.032781917s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X..X..X.X..............X....................................] duration: 5.785259927s FAIL (2025-07-27 01:21:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.78527679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXXX..X.X.X..........X...................................] duration: 5.767083891s FAIL (2025-07-27 01:21:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.767110281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXXX.......XXXXXX......................................] duration: 5.818918835s FAIL (2025-07-27 01:21:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.818944113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XXXX.....X....X....XXX....................................] duration: 5.715824617s FAIL (2025-07-27 01:21:10 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.715850828s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [XX..X..X.X..............X...................................] duration: 5.627888421s FAIL (2025-07-27 01:21:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.627911364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXX.......XXXXXX.....................................] duration: 5.791439872s FAIL (2025-07-27 01:21:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.791465851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....................X....X..........X...................X..] duration: 2.05220482s FAIL (2025-07-27 01:20:45 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 2.052234015s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................X....X..........X...................X.] duration: 1.877738013s FAIL (2025-07-27 01:20:29 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 1.877762281s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXXXX..X.X.X..........X..................................] duration: 5.74573858s FAIL (2025-07-27 01:20:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.74578028s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXXX.......XXXXXX....................................] duration: 5.7679095s FAIL (2025-07-27 01:20:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.76793512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [X..XXXX.....X....X....XXX...................................] duration: 5.812264353s FAIL (2025-07-27 01:19:56 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.812290462s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......................X....X..........X...................X] duration: 2.831176485s FAIL (2025-07-27 01:19:56 UTC) orbiter warning: digest mismatch: after 792f8e52f0f261131431c48786048ce73833f71a202ca313ecdd5b39aa603e1c != before 4c0418a6860f4eb95c9171181a1c1a28de1d7886deefc8fd7b2ed1a91cded834 Duration: 2.831199167s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [XXX..X..X.X..............X..................................] duration: 5.863340678s FAIL (2025-07-27 01:19:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.863385673s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.......XXXXXX...................................] duration: 5.891007195s FAIL (2025-07-27 01:19:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.891040277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XXXXXXXX..X.X.X..........X.................................] duration: 5.935600181s FAIL (2025-07-27 01:19:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.935652641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX..XXXX.....X....X....XXX..................................] duration: 5.748858088s FAIL (2025-07-27 01:19:25 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.748883558s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.XXX..X..X.X..............X.................................] duration: 5.552983312s FAIL (2025-07-27 01:19:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.553006575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.......XXXXXX..................................] duration: 5.537043739s FAIL (2025-07-27 01:19:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.53707082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.XXXXXXXX..X.X.X..........X................................] duration: 5.763622297s FAIL (2025-07-27 01:19:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.763645252s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................X....X..........X...................] duration: 1.994506645s FAIL (2025-07-27 01:18:58 UTC) orbiter warning: digest mismatch: after 0fab96406578ff4f6e5ac3bcbf27e9ffb66f17d6676b38ac64e45ce1eebe95f6 != before 0f0419772558bb804c8dac0f8b10ef5a33d3e2f1a02a6ab2fe9f121ad674c5d9 Duration: 1.994534338s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..XXX..X..X.X..............X................................] duration: 5.597464699s FAIL (2025-07-27 01:18:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.597484137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXX..XXXX.....X....X....XXX.................................] duration: 7.481649665s FAIL (2025-07-27 01:18:42 UTC) orbiter warning: uplink: context deadline exceeded Duration: 7.4816681s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.......XXXXXX.................................] duration: 5.554131449s FAIL (2025-07-27 01:18:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.554154383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXXX..XXXX.....X....X....XXX................................] duration: 9.265662323s FAIL (2025-07-27 01:18:19 UTC) orbiter warning: uplink: context deadline exceeded Duration: 9.26568749s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X..XXX..X..X.X..............X...............................] duration: 5.724712871s FAIL (2025-07-27 01:18:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.724738511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX.XXXXXXXX..X.X.X..........X...............................] duration: 5.703795451s FAIL (2025-07-27 01:17:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.703869922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................X....X..........X..................] duration: 1.833741593s FAIL (2025-07-27 01:17:56 UTC) orbiter warning: digest mismatch: after 0fab96406578ff4f6e5ac3bcbf27e9ffb66f17d6676b38ac64e45ce1eebe95f6 != before 0f0419772558bb804c8dac0f8b10ef5a33d3e2f1a02a6ab2fe9f121ad674c5d9 Duration: 1.833786139s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXXX..XXXX.....X....X....XXX...............................] duration: 5.994138221s FAIL (2025-07-27 01:17:53 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.99416406s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.......XXXXXX................................] duration: 5.541403907s FAIL (2025-07-27 01:17:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.541433895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.XXXX..XXXX.....X....X....XXX..............................] duration: 5.857473532s FAIL (2025-07-27 01:17:33 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.857495225s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....X..........X.................] duration: 1.843796211s FAIL (2025-07-27 01:17:31 UTC) orbiter warning: digest mismatch: after 0fab96406578ff4f6e5ac3bcbf27e9ffb66f17d6676b38ac64e45ce1eebe95f6 != before 0f0419772558bb804c8dac0f8b10ef5a33d3e2f1a02a6ab2fe9f121ad674c5d9 Duration: 1.843826279s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.......XXXXXX...............................] duration: 8.036085374s FAIL (2025-07-27 01:17:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 8.036115272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.950666627s FAIL (2025-07-27 01:17:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.950686796s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXX.XXXXXXXX..X.X.X..........X..............................] duration: 6.876923766s FAIL (2025-07-27 01:17:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.876942552s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX.XXXX..XXXX.....X....X....XXX.............................] duration: 22.645642377s FAIL (2025-07-27 01:17:16 UTC) orbiter warning: uplink: context deadline exceeded Duration: 22.645666664s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [XX..XXX..X..X.X..............X..............................] duration: 6.67692217s FAIL (2025-07-27 01:16:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.676944723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....X..........X................] duration: 1.865211435s FAIL (2025-07-27 01:16:38 UTC) orbiter warning: digest mismatch: after 0fab96406578ff4f6e5ac3bcbf27e9ffb66f17d6676b38ac64e45ce1eebe95f6 != before 0f0419772558bb804c8dac0f8b10ef5a33d3e2f1a02a6ab2fe9f121ad674c5d9 Duration: 1.865238718s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.......XXXXXX..............................] duration: 5.685224419s FAIL (2025-07-27 01:16:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.685245119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXX.XXXXXXXX..X.X.X..........X.............................] duration: 5.738476096s FAIL (2025-07-27 01:16:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.738504571s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 6.283740999s FAIL (2025-07-27 01:16:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.283761719s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................X....X..........X...............] duration: 1.863180699s FAIL (2025-07-27 01:16:21 UTC) orbiter warning: digest mismatch: after 0fab96406578ff4f6e5ac3bcbf27e9ffb66f17d6676b38ac64e45ce1eebe95f6 != before 0f0419772558bb804c8dac0f8b10ef5a33d3e2f1a02a6ab2fe9f121ad674c5d9 Duration: 1.863203882s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXX.XXXX..XXXX.....X....X....XXX............................] duration: 5.793072465s FAIL (2025-07-27 01:16:17 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.793099887s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [XXX..XXX..X..X.X..............X.............................] duration: 5.678239642s FAIL (2025-07-27 01:15:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.678263599s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 6.177812717s FAIL (2025-07-27 01:15:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.177839337s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)