Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.X..........X....] duration: 5.992416857s FAIL (2025-07-28 00:26:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== M3CTHVHBOICECUDARKRK2KDCQAJ4GZUI543EMR72LUNYHSU62IWA, seg_pos={0 0}, upload=81361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Jul 28 00:22:41.678787344 (84h46m51.401350384s) [seg_pos={0 0}, upload=81361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:22:41.678874520 (84h46m51.40143756s) [upload=81361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:22:41.678879389 (84h46m51.401442429s) [upload=81361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:22:41.957215845 (84h46m51.679778875s) [upload=81361]: Done uploading segments... Jul 28 00:22:41.957233678 (84h46m51.679796718s) [upload=81361]: Upload finished. err: <nil> Jul 28 00:23:37.076289486 (84h47m46.798852536s) [upload=81374]: Starting upload Jul 28 00:23:37.076409375 (84h47m46.798972425s) [upload=81374]: Done starting upload Jul 28 00:23:37.076524956 (84h47m46.799087996s) [upload=81374]: Uploading segments... Jul 28 00:23:37.076610781 (84h47m46.799173821s) [upload=81374]: Got next segment. Inline: false Jul 28 00:23:37.076641660 (84h47m46.7992047s) [upload=81374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:23:37.076646429 (84h47m46.799209459s) [upload=81374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:23:37.317328312 (84h47m47.039891352s) [seg_pos={0 0}, upload=81374]: Begin upload segment... Jul 28 00:23:37.317375502 (84h47m47.039938542s) [seg_pos={0 0}, upload=81374]: Done begin upload segment. Jul 28 00:23:37.317387365 (84h47m47.039950405s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317398867 (84h47m47.039961907s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317741793 (84h47m47.040304833s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317771279 (84h47m47.040334329s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318039613 (84h47m47.040602653s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318294580 (84h47m47.04085762s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318532536 (84h47m47.041095566s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318775812 (84h47m47.041338852s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318964052 (84h47m47.041527092s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319152892 (84h47m47.041715932s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319350842 (84h47m47.041913882s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319540073 (84h47m47.042103113s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319723063 (84h47m47.042286094s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319905383 (84h47m47.042468423s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320084705 (84h47m47.042647745s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320267355 (84h47m47.042830395s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:39.064964023 (84h47m48.787527053s) [upload=81374]: Next returned nil segment Jul 28 00:23:39.064981687 (84h47m48.787544727s) [upload=81374]: Waiting for error group managing segments... Jul 28 00:23:39.340867483 (84h47m49.063430523s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.356965262 (84h47m49.079528302s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.362359653 (84h47m49.084922693s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.373141581 (84h47m49.095704651s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.430640531 (84h47m49.153203571s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.445133640 (84h47m49.16769667s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.454045370 (84h47m49.1766084s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.503182252 (84h47m49.225745292s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.519937578 (84h47m49.242500618s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.522381895 (84h47m49.244944935s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.531306760 (84h47m49.2538698s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.602626742 (84h47m49.325189792s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:24:06.434846113 (84h48m16.157409153s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:45336->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434872944 (84h48m16.157435984s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434917430 (84h48m16.15748047s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:60368->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434930544 (84h48m16.157493584s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:55716->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434955502 (84h48m16.157518542s) [seg_pos={0 0}, upload=81374]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:24:06.663297936 (84h48m16.385860976s) [upload=81374]: Done uploading segments... Jul 28 00:24:06.663325118 (84h48m16.385888158s) [upload=81374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.992444391s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................X............X....................] duration: 1.939475266s FAIL (2025-07-28 00:26:07 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.93950965s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.......................................X...XXXXXXXXXXXXXXXXX] duration: 1m0.236474257s FAIL (2025-07-28 00:25:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 1 , piece_id=VTDXU26RJTFYH7AMTBQZWQ3QKYVDZFNHPOCJHFRDMH5AQ2FULMRA, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.220321220 (84h49m9.94288427s) [noise=true , address=104.36.82.50:4023 , piece_id=QPF4KEWUDQJTIQRTDT7S5SWXHFOYGKKQURSMN4WBCX24M6GJQ6TA, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.222382443 (84h49m9.944945483s) [noise=true , address=104.36.82.15:4269 , piece_id=CDRX7KNIW75FVJWVXLGMM2A2MYOIA36I3QSAXEMOOMOD6KWRNYVQ, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.223240054 (84h49m9.945803094s) [noise=true , address=79.127.201.215:4353 , piece_id=3S6Q5RBJ7CUNW6DOJ4RDYWMFBKV4T5MYSZTWTCUHWJR7IML4TSRA, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.223517074 (84h49m9.946080114s) [noise=true , address=104.36.82.16:4316 , piece_id=J6BSARM5TOFVMLF6HI2DCZLX3EM4XXOJ4DS2OMIOSMVQW6HRXTOA, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.223786409 (84h49m9.946349449s) [noise=true , address=104.36.82.54:4172 , piece_id=34MW52WES675ULL7RPI27KJFS4IYXG7QSM4OTGWU7SYPDDIBGVXQ, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.224019285 (84h49m9.946582325s) [noise=true , address=64.49.13.138:4482 , piece_id=ETGNED6LEFUADOW2QLXHNG2SUDC4ZBD5HLS5Y5EJTRMQXVM3IV4A, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:00.224236911 (84h49m9.946799951s) [noise=true , address=79.127.205.235:4669 , piece_id=6AFZVGHOP6KZJPECD2UIW33B6F6Q26AA5ZIC4BW7UWO2J7FZ66WQ, seg_pos={0 0}, upload=81398]: Uploading piece... Jul 28 00:25:01.277058093 (84h49m10.999621133s) [noise=true , address=64.49.13.138:4482 , piece_id=ETGNED6LEFUADOW2QLXHNG2SUDC4ZBD5HLS5Y5EJTRMQXVM3IV4A, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 28 00:25:01.349103484 (84h49m11.071666524s) [upload=81398]: Next returned nil segment Jul 28 00:25:01.349124555 (84h49m11.071687625s) [upload=81398]: Waiting for error group managing segments... Jul 28 00:25:01.664316903 (84h49m11.386879953s) [noise=true , address=79.127.205.235:4669 , piece_id=6AFZVGHOP6KZJPECD2UIW33B6F6Q26AA5ZIC4BW7UWO2J7FZ66WQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: <nil> Jul 28 00:25:03.593833469 (84h49m13.316396519s) [noise=true , address=79.127.201.215:4353 , piece_id=3S6Q5RBJ7CUNW6DOJ4RDYWMFBKV4T5MYSZTWTCUHWJR7IML4TSRA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: <nil> Jul 28 00:25:59.128138643 (84h50m8.850701683s) [noise=true , address=104.36.82.55:4193 , piece_id=C2AD5AHRRCVIOB6NF3T2JBDULPMDFGIT3WKYNLVSUZDEDPZY4CUQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:104.36.82.55:4193): protocol: write tcp 188.166.225.170:54014->104.36.82.55:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128195162 (84h50m8.850758202s) [noise=true , address=104.36.82.53:4133 , piece_id=UAPKBBUFJEA2ZZ423UPHCXVTQRFGHUG3GGYKULYYN4OIY4XJC4VA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: write tcp 188.166.225.170:51602->104.36.82.53:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128230790 (84h50m8.85079383s) [noise=true , address=104.36.82.47:4524 , piece_id=BPLN3IAWLKWDARSLQI6KV55VZCONMK466LDBNPBZ6WBJV6FZGGRQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 188.166.225.170:48814->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128281688 (84h50m8.850844718s) [noise=true , address=104.36.82.16:4316 , piece_id=J6BSARM5TOFVMLF6HI2DCZLX3EM4XXOJ4DS2OMIOSMVQW6HRXTOA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12PPnXDzwXDVDKemw483GXuzQrPN48JJrjeeD7az2NPNawHNaXf, address:104.36.82.16:4316): protocol: rpc: dial tcp 104.36.82.16:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128318889 (84h50m8.850881929s) [noise=true , address=104.36.82.14:4231 , piece_id=FO6CDNXEKQ6H6QZYRY2FIWR7DVHKX56CSUAMELSDL5ZGJXLGKH3A, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1ixY5pokE6MtrVH9FpKc1tGZhEAaW5brbzBeeGmYUczK3z7tCQ, address:104.36.82.14:4231): protocol: write tcp 188.166.225.170:40198->104.36.82.14:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128397740 (84h50m8.85096078s) [noise=true , address=104.36.82.4:4833 , piece_id=3MVB6JNGMHQPR4W4C4OPNBLYG66YOXCI6ZOT5AB5O3OEOSI5CQOA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: write tcp 188.166.225.170:54030->104.36.82.4:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128450030 (84h50m8.85101307s) [noise=true , address=104.36.82.35:4589 , piece_id=POFSWYNKJMSV6W3CWKW32XCCI5JCEYCU3TJKYWUEREK4CJZRTHVQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: write tcp 188.166.225.170:51890->104.36.82.35:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128503592 (84h50m8.851066632s) [noise=true , address=104.36.82.52:4107 , piece_id=6MRSKLF5ZMDEXJU4PJPP5MT5KHG7AZXTQYEHJ4STYUAZOHOXVSKQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: write tcp 188.166.225.170:40086->104.36.82.52:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128556944 (84h50m8.851119995s) [noise=true , address=104.36.82.12:4171 , piece_id=VTDXU26RJTFYH7AMTBQZWQ3QKYVDZFNHPOCJHFRDMH5AQ2FULMRA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1z2Fh5fPFTb9VZX6to5hdUYUGst63Ti1BVtdVtYk7vkrUqXMKM, address:104.36.82.12:4171): protocol: write tcp 188.166.225.170:45684->104.36.82.12:4171: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128600056 (84h50m8.851163096s) [noise=true , address=104.36.82.54:4172 , piece_id=34MW52WES675ULL7RPI27KJFS4IYXG7QSM4OTGWU7SYPDDIBGVXQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:104.36.82.54:4172): protocol: rpc: dial tcp 104.36.82.54:4172: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128632218 (84h50m8.851195248s) [noise=true , address=104.36.82.50:4023 , piece_id=QPF4KEWUDQJTIQRTDT7S5SWXHFOYGKKQURSMN4WBCX24M6GJQ6TA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12c8EYHav34x4uWw8LnwFZqMdDTfoJDzJS3K6rA8dAQv6nHrK4v, address:104.36.82.50:4023): protocol: rpc: dial tcp 104.36.82.50:4023: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128636537 (84h50m8.851199577s) [noise=true , address=104.36.82.13:4218 , piece_id=R4XMR6N3SENNU7EXLWVIZ7TLLEELZJ3CEMDBYN7PTAC4SOH7AS2A, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc, address:104.36.82.13:4218): protocol: write tcp 188.166.225.170:56396->104.36.82.13:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128676383 (84h50m8.851239423s) [noise=true , address=104.36.82.46:4489 , piece_id=VLIPR6A2NXKAEEWSXSGOEXPTFAOHFOTE2EAYJQN4CBBCMV2Z74DA, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: write tcp 188.166.225.170:49202->104.36.82.46:4489: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128683456 (84h50m8.851246496s) [noise=true , address=104.36.82.45:4906 , piece_id=Z5VX4AFLM72RVM26K5KOC6NM7IGHTTR2ZLWAF6YUIPW6JRAEOD5A, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:104.36.82.45:4906): protocol: write tcp 188.166.225.170:46586->104.36.82.45:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128705598 (84h50m8.851268639s) [noise=true , address=104.36.82.15:4269 , piece_id=CDRX7KNIW75FVJWVXLGMM2A2MYOIA36I3QSAXEMOOMOD6KWRNYVQ, seg_pos={0 0}, upload=81398]: Done uploading piece. err: ecclient: upload failed (node:12V8v5nzyyouSqndATi4fRJUC6HgFbhguBjnmwpki48zdRBTcgC, address:104.36.82.15:4269): protocol: rpc: dial tcp 104.36.82.15:4269: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:59.128785031 (84h50m8.851348071s) [seg_pos={0 0}, upload=81398]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:25:59.362015718 (84h50m9.084578758s) [upload=81398]: Done uploading segments... Jul 28 00:25:59.362039173 (84h50m9.084602213s) [upload=81398]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.236488885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.X..........X...] duration: 5.767887802s FAIL (2025-07-28 00:25:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== M3CTHVHBOICECUDARKRK2KDCQAJ4GZUI543EMR72LUNYHSU62IWA, seg_pos={0 0}, upload=81361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Jul 28 00:22:41.678787344 (84h46m51.401350384s) [seg_pos={0 0}, upload=81361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:22:41.678874520 (84h46m51.40143756s) [upload=81361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:22:41.678879389 (84h46m51.401442429s) [upload=81361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:22:41.957215845 (84h46m51.679778875s) [upload=81361]: Done uploading segments... Jul 28 00:22:41.957233678 (84h46m51.679796718s) [upload=81361]: Upload finished. err: <nil> Jul 28 00:23:37.076289486 (84h47m46.798852536s) [upload=81374]: Starting upload Jul 28 00:23:37.076409375 (84h47m46.798972425s) [upload=81374]: Done starting upload Jul 28 00:23:37.076524956 (84h47m46.799087996s) [upload=81374]: Uploading segments... Jul 28 00:23:37.076610781 (84h47m46.799173821s) [upload=81374]: Got next segment. Inline: false Jul 28 00:23:37.076641660 (84h47m46.7992047s) [upload=81374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:23:37.076646429 (84h47m46.799209459s) [upload=81374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:23:37.317328312 (84h47m47.039891352s) [seg_pos={0 0}, upload=81374]: Begin upload segment... Jul 28 00:23:37.317375502 (84h47m47.039938542s) [seg_pos={0 0}, upload=81374]: Done begin upload segment. Jul 28 00:23:37.317387365 (84h47m47.039950405s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317398867 (84h47m47.039961907s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317741793 (84h47m47.040304833s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317771279 (84h47m47.040334329s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318039613 (84h47m47.040602653s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318294580 (84h47m47.04085762s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318532536 (84h47m47.041095566s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318775812 (84h47m47.041338852s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318964052 (84h47m47.041527092s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319152892 (84h47m47.041715932s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319350842 (84h47m47.041913882s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319540073 (84h47m47.042103113s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319723063 (84h47m47.042286094s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319905383 (84h47m47.042468423s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320084705 (84h47m47.042647745s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320267355 (84h47m47.042830395s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:39.064964023 (84h47m48.787527053s) [upload=81374]: Next returned nil segment Jul 28 00:23:39.064981687 (84h47m48.787544727s) [upload=81374]: Waiting for error group managing segments... Jul 28 00:23:39.340867483 (84h47m49.063430523s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.356965262 (84h47m49.079528302s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.362359653 (84h47m49.084922693s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.373141581 (84h47m49.095704651s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.430640531 (84h47m49.153203571s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.445133640 (84h47m49.16769667s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.454045370 (84h47m49.1766084s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.503182252 (84h47m49.225745292s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.519937578 (84h47m49.242500618s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.522381895 (84h47m49.244944935s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.531306760 (84h47m49.2538698s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.602626742 (84h47m49.325189792s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:24:06.434846113 (84h48m16.157409153s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:45336->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434872944 (84h48m16.157435984s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434917430 (84h48m16.15748047s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:60368->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434930544 (84h48m16.157493584s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:55716->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434955502 (84h48m16.157518542s) [seg_pos={0 0}, upload=81374]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:24:06.663297936 (84h48m16.385860976s) [upload=81374]: Done uploading segments... Jul 28 00:24:06.663325118 (84h48m16.385888158s) [upload=81374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.767917257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................X.......X.XX....X] duration: 30.247971673s FAIL (2025-07-28 00:25:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.247990067s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X.X..........X..] duration: 6.304131053s FAIL (2025-07-28 00:25:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== M3CTHVHBOICECUDARKRK2KDCQAJ4GZUI543EMR72LUNYHSU62IWA, seg_pos={0 0}, upload=81361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Jul 28 00:22:41.678787344 (84h46m51.401350384s) [seg_pos={0 0}, upload=81361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:22:41.678874520 (84h46m51.40143756s) [upload=81361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:22:41.678879389 (84h46m51.401442429s) [upload=81361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:22:41.957215845 (84h46m51.679778875s) [upload=81361]: Done uploading segments... Jul 28 00:22:41.957233678 (84h46m51.679796718s) [upload=81361]: Upload finished. err: <nil> Jul 28 00:23:37.076289486 (84h47m46.798852536s) [upload=81374]: Starting upload Jul 28 00:23:37.076409375 (84h47m46.798972425s) [upload=81374]: Done starting upload Jul 28 00:23:37.076524956 (84h47m46.799087996s) [upload=81374]: Uploading segments... Jul 28 00:23:37.076610781 (84h47m46.799173821s) [upload=81374]: Got next segment. Inline: false Jul 28 00:23:37.076641660 (84h47m46.7992047s) [upload=81374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:23:37.076646429 (84h47m46.799209459s) [upload=81374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:23:37.317328312 (84h47m47.039891352s) [seg_pos={0 0}, upload=81374]: Begin upload segment... Jul 28 00:23:37.317375502 (84h47m47.039938542s) [seg_pos={0 0}, upload=81374]: Done begin upload segment. Jul 28 00:23:37.317387365 (84h47m47.039950405s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317398867 (84h47m47.039961907s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317741793 (84h47m47.040304833s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317771279 (84h47m47.040334329s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318039613 (84h47m47.040602653s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318294580 (84h47m47.04085762s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318532536 (84h47m47.041095566s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318775812 (84h47m47.041338852s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318964052 (84h47m47.041527092s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319152892 (84h47m47.041715932s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319350842 (84h47m47.041913882s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319540073 (84h47m47.042103113s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319723063 (84h47m47.042286094s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319905383 (84h47m47.042468423s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320084705 (84h47m47.042647745s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320267355 (84h47m47.042830395s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:39.064964023 (84h47m48.787527053s) [upload=81374]: Next returned nil segment Jul 28 00:23:39.064981687 (84h47m48.787544727s) [upload=81374]: Waiting for error group managing segments... Jul 28 00:23:39.340867483 (84h47m49.063430523s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.356965262 (84h47m49.079528302s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.362359653 (84h47m49.084922693s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.373141581 (84h47m49.095704651s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.430640531 (84h47m49.153203571s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.445133640 (84h47m49.16769667s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.454045370 (84h47m49.1766084s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.503182252 (84h47m49.225745292s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.519937578 (84h47m49.242500618s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.522381895 (84h47m49.244944935s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.531306760 (84h47m49.2538698s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.602626742 (84h47m49.325189792s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:24:06.434846113 (84h48m16.157409153s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:45336->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434872944 (84h48m16.157435984s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434917430 (84h48m16.15748047s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:60368->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434930544 (84h48m16.157493584s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:55716->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434955502 (84h48m16.157518542s) [seg_pos={0 0}, upload=81374]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:24:06.663297936 (84h48m16.385860976s) [upload=81374]: Done uploading segments... Jul 28 00:24:06.663325118 (84h48m16.385888158s) [upload=81374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.304159056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................X............X...................] duration: 1.869113639s FAIL (2025-07-28 00:24:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.869138958s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [........................................X...XXXXXXXXXXXXXXXX] duration: 1m0.231620237s FAIL (2025-07-28 00:24:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== . Jul 28 00:23:59.802230820 (84h48m9.52479386s) [noise=true , address=104.36.82.11:4138 , piece_id=GTFDXI4VZGCDJRWPY3OKK3JUYBSWQ5YCOKW5C3SMUAJI4VI7222Q, seg_pos={0 0}, upload=81383]: Uploading piece... Jul 28 00:23:59.802578515 (84h48m9.525141556s) [noise=true , address=104.36.82.51:4074 , piece_id=QBUOGL7FFHACOQERFGDWPCHXLA6ED6ERJ35AF6QJKFMYYOUWMPHA, seg_pos={0 0}, upload=81383]: Uploading piece... Jul 28 00:23:59.803188272 (84h48m9.525751312s) [noise=true , address=104.36.82.10:4100 , piece_id=B4AWMN4AXSEGFVYVS6TJKCWRILV6BCGXEQJ246HO3BGPBJKGOW6A, seg_pos={0 0}, upload=81383]: Uploading piece... Jul 28 00:23:59.803643714 (84h48m9.526206744s) [noise=true , address=104.36.82.12:4184 , piece_id=7QROS7F572LUU7W6CIISRT45G3WCPZBLNDXEN52EARGS4OZBCXUA, seg_pos={0 0}, upload=81383]: Uploading piece... Jul 28 00:23:59.803929851 (84h48m9.526492891s) [noise=true , address=104.36.82.14:4247 , piece_id=MYXMQISD5GZGRYQZQ6BELILN725JU3E4EKQGDHSZJYLKNR6DE4QA, seg_pos={0 0}, upload=81383]: Uploading piece... Jul 28 00:23:59.805421925 (84h48m9.527984955s) [noise=true , address=104.36.82.55:4196 , piece_id=FQMBMD6RXULAGOP3PSMM2T57IYUIGBVVEOYWJUOTOPO5RY4GB7KQ, seg_pos={0 0}, upload=81383]: Uploading piece... Jul 28 00:24:01.291154224 (84h48m11.013717284s) [upload=81383]: Next returned nil segment Jul 28 00:24:01.291179543 (84h48m11.013742593s) [upload=81383]: Waiting for error group managing segments... Jul 28 00:24:06.713031350 (84h48m16.43559438s) [noise=true , address=79.127.201.215:4352 , piece_id=ABEAQ566ERILLAS37ERYHVTOMLRL37GFB7AEWHEUYRGUQ3HVTQNA, seg_pos={0 0}, upload=81383]: Done uploading piece. err: <nil> Jul 28 00:24:58.663550008 (84h49m8.386113048s) [noise=true , address=104.36.82.12:4184 , piece_id=7QROS7F572LUU7W6CIISRT45G3WCPZBLNDXEN52EARGS4OZBCXUA, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:17JJjwwgSn7A6BS7n1a7fphX8oTEQpXmad25ifM2nDhWhPv4Ls, address:104.36.82.12:4184): protocol: rpc: dial tcp 104.36.82.12:4184: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.663655820 (84h49m8.38621885s) [noise=true , address=104.36.82.42:4818 , piece_id=SZ4RVMUSHN3KQRA76HBQJLBDTE6L3J65TVSHNHYYJERCOX7AE7KA, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:104.36.82.42:4818): protocol: write tcp 188.166.225.170:33386->104.36.82.42:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.663712278 (84h49m8.386275318s) [noise=true , address=104.36.82.15:4268 , piece_id=VRAAGUWFRGRQWNZMPWGFY26VSFNNQ6OS5LH45RQXCLCQ7JYJGWHA, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:112ocg8g8hf2huUskC1Kty3z51wXc8VuNUgARTFjKZxsF6RFFy, address:104.36.82.15:4268): protocol: write tcp 188.166.225.170:35638->104.36.82.15:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.663744470 (84h49m8.38630751s) [noise=true , address=104.36.82.53:4129 , piece_id=PHKL22EOXHXN7WO57T5JPEHCIUSNZB2WEA53GHHJR4ONXEOD27DQ, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: rpc: dial tcp 104.36.82.53:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.663801369 (84h49m8.38636441s) [noise=true , address=104.36.82.51:4074 , piece_id=QBUOGL7FFHACOQERFGDWPCHXLA6ED6ERJ35AF6QJKFMYYOUWMPHA, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc, address:104.36.82.51:4074): protocol: write tcp 188.166.225.170:60546->104.36.82.51:4074: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664049174 (84h49m8.386612214s) [noise=true , address=104.36.82.54:4179 , piece_id=TO3VPL4O4JSNA7J75Z3AS2GT7YYAI57BAZJX6DXK2M5LYKHF2MRQ, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP, address:104.36.82.54:4179): protocol: write tcp 188.166.225.170:46530->104.36.82.54:4179: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664083709 (84h49m8.386646749s) [noise=true , address=104.36.82.21:4482 , piece_id=KMVJY2IVJHBACXRW4OLBD7CGEXCYGYP6Q7UMKZBM7KU7DDKHR26Q, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:12B2s428CMVK7mdLV84VgWFcdRQaVQxcA7YuqqeqEM8FAefEGt7, address:104.36.82.21:4482): protocol: write tcp 188.166.225.170:44694->104.36.82.21:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664109369 (84h49m8.386672409s) [noise=true , address=104.36.82.11:4138 , piece_id=GTFDXI4VZGCDJRWPY3OKK3JUYBSWQ5YCOKW5C3SMUAJI4VI7222Q, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: write tcp 188.166.225.170:48024->104.36.82.11:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664136921 (84h49m8.386699961s) [noise=true , address=104.36.82.10:4100 , piece_id=B4AWMN4AXSEGFVYVS6TJKCWRILV6BCGXEQJ246HO3BGPBJKGOW6A, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:12Hyc6sbtuPXMLyFzubMcepbJyLyVAGNpkXAmqg73i3sY97dwdW, address:104.36.82.10:4100): protocol: write tcp 188.166.225.170:49410->104.36.82.10:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664146219 (84h49m8.386709259s) [noise=true , address=104.36.82.20:4468 , piece_id=P7XMH62YQX24DNUGQOFJVVPUK2PZC2Y7DRJTQLHG6RYYMPEOE33Q, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:4468): protocol: write tcp 188.166.225.170:55022->104.36.82.20:4468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664166578 (84h49m8.386729618s) [noise=true , address=104.36.82.55:4196 , piece_id=FQMBMD6RXULAGOP3PSMM2T57IYUIGBVVEOYWJUOTOPO5RY4GB7KQ, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: rpc: dial tcp 104.36.82.55:4196: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664191305 (84h49m8.386754355s) [noise=true , address=104.36.82.45:4932 , piece_id=EUCFMMBS5LON45RQPQVZOAOW2VVVXSPGO5T3CLHRFWTEP4VAWZ5Q, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm, address:104.36.82.45:4932): protocol: write tcp 188.166.225.170:44084->104.36.82.45:4932: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664225470 (84h49m8.386788511s) [noise=true , address=104.36.82.40:4950 , piece_id=VSI4PYGRNEF65E4R4BDQAH3ISWPW7T6ZQYBUJN5L7UEFOGJDCA3Q, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk, address:104.36.82.40:4950): protocol: rpc: dial tcp 104.36.82.40:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664231502 (84h49m8.386794542s) [noise=true , address=104.36.82.46:4507 , piece_id=HPSNTI7ZNXSTFTZCII5EFG75QM5RWIHAT5JGSWB4LHLB5A7ZWGOQ, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: write tcp 188.166.225.170:57976->104.36.82.46:4507: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664278101 (84h49m8.386841141s) [noise=true , address=104.36.82.9:4026 , piece_id=WJ6KVFGXXWYJRRMQYTQD2K6E3VJBRWHXLKSVBBSCYRKNSQN23VRQ, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:1M7HJBLoxHkRHXbPuDiqwxdb4EAq5wktfJY5FygdoDdgy59rRq, address:104.36.82.9:4026): protocol: write tcp 188.166.225.170:57466->104.36.82.9:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664283892 (84h49m8.386846932s) [noise=true , address=104.36.82.14:4247 , piece_id=MYXMQISD5GZGRYQZQ6BELILN725JU3E4EKQGDHSZJYLKNR6DE4QA, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:104.36.82.14:4247): protocol: rpc: dial tcp 104.36.82.14:4247: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664295494 (84h49m8.386858534s) [noise=true , address=104.36.82.13:4220 , piece_id=7SCONB4RSFHJJ5QTA6HF7CLLD3LSSVLQJWHS6DR3OUILYAPK7WHQ, seg_pos={0 0}, upload=81383]: Done uploading piece. err: ecclient: upload failed (node:1AjQmw3H58hTwr2389cmMcwy1gBavBrZB8AWB1gqfoFRUkCpH6, address:104.36.82.13:4220): protocol: write tcp 188.166.225.170:40280->104.36.82.13:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:58.664316975 (84h49m8.386880015s) [seg_pos={0 0}, upload=81383]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:24:58.893945261 (84h49m8.616508301s) [upload=81383]: Done uploading segments... Jul 28 00:24:58.893965399 (84h49m8.616528439s) [upload=81383]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.231631619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X.X..........X.] duration: 5.962202014s FAIL (2025-07-28 00:24:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== M3CTHVHBOICECUDARKRK2KDCQAJ4GZUI543EMR72LUNYHSU62IWA, seg_pos={0 0}, upload=81361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Jul 28 00:22:41.678787344 (84h46m51.401350384s) [seg_pos={0 0}, upload=81361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:22:41.678874520 (84h46m51.40143756s) [upload=81361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:22:41.678879389 (84h46m51.401442429s) [upload=81361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:22:41.957215845 (84h46m51.679778875s) [upload=81361]: Done uploading segments... Jul 28 00:22:41.957233678 (84h46m51.679796718s) [upload=81361]: Upload finished. err: <nil> Jul 28 00:23:37.076289486 (84h47m46.798852536s) [upload=81374]: Starting upload Jul 28 00:23:37.076409375 (84h47m46.798972425s) [upload=81374]: Done starting upload Jul 28 00:23:37.076524956 (84h47m46.799087996s) [upload=81374]: Uploading segments... Jul 28 00:23:37.076610781 (84h47m46.799173821s) [upload=81374]: Got next segment. Inline: false Jul 28 00:23:37.076641660 (84h47m46.7992047s) [upload=81374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:23:37.076646429 (84h47m46.799209459s) [upload=81374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:23:37.317328312 (84h47m47.039891352s) [seg_pos={0 0}, upload=81374]: Begin upload segment... Jul 28 00:23:37.317375502 (84h47m47.039938542s) [seg_pos={0 0}, upload=81374]: Done begin upload segment. Jul 28 00:23:37.317387365 (84h47m47.039950405s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317398867 (84h47m47.039961907s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317741793 (84h47m47.040304833s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317771279 (84h47m47.040334329s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318039613 (84h47m47.040602653s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318294580 (84h47m47.04085762s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318532536 (84h47m47.041095566s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318775812 (84h47m47.041338852s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318964052 (84h47m47.041527092s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319152892 (84h47m47.041715932s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319350842 (84h47m47.041913882s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319540073 (84h47m47.042103113s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319723063 (84h47m47.042286094s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319905383 (84h47m47.042468423s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320084705 (84h47m47.042647745s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320267355 (84h47m47.042830395s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:39.064964023 (84h47m48.787527053s) [upload=81374]: Next returned nil segment Jul 28 00:23:39.064981687 (84h47m48.787544727s) [upload=81374]: Waiting for error group managing segments... Jul 28 00:23:39.340867483 (84h47m49.063430523s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.356965262 (84h47m49.079528302s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.362359653 (84h47m49.084922693s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.373141581 (84h47m49.095704651s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.430640531 (84h47m49.153203571s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.445133640 (84h47m49.16769667s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.454045370 (84h47m49.1766084s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.503182252 (84h47m49.225745292s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.519937578 (84h47m49.242500618s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.522381895 (84h47m49.244944935s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.531306760 (84h47m49.2538698s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.602626742 (84h47m49.325189792s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:24:06.434846113 (84h48m16.157409153s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:45336->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434872944 (84h48m16.157435984s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434917430 (84h48m16.15748047s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:60368->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434930544 (84h48m16.157493584s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:55716->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434955502 (84h48m16.157518542s) [seg_pos={0 0}, upload=81374]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:24:06.663297936 (84h48m16.385860976s) [upload=81374]: Done uploading segments... Jul 28 00:24:06.663325118 (84h48m16.385888158s) [upload=81374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.962227352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X.X..........X] duration: 30.230898931s FAIL (2025-07-28 00:24:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== M3CTHVHBOICECUDARKRK2KDCQAJ4GZUI543EMR72LUNYHSU62IWA, seg_pos={0 0}, upload=81361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Jul 28 00:22:41.678787344 (84h46m51.401350384s) [seg_pos={0 0}, upload=81361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:22:41.678874520 (84h46m51.40143756s) [upload=81361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:22:41.678879389 (84h46m51.401442429s) [upload=81361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:22:41.957215845 (84h46m51.679778875s) [upload=81361]: Done uploading segments... Jul 28 00:22:41.957233678 (84h46m51.679796718s) [upload=81361]: Upload finished. err: <nil> Jul 28 00:23:37.076289486 (84h47m46.798852536s) [upload=81374]: Starting upload Jul 28 00:23:37.076409375 (84h47m46.798972425s) [upload=81374]: Done starting upload Jul 28 00:23:37.076524956 (84h47m46.799087996s) [upload=81374]: Uploading segments... Jul 28 00:23:37.076610781 (84h47m46.799173821s) [upload=81374]: Got next segment. Inline: false Jul 28 00:23:37.076641660 (84h47m46.7992047s) [upload=81374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:23:37.076646429 (84h47m46.799209459s) [upload=81374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:23:37.317328312 (84h47m47.039891352s) [seg_pos={0 0}, upload=81374]: Begin upload segment... Jul 28 00:23:37.317375502 (84h47m47.039938542s) [seg_pos={0 0}, upload=81374]: Done begin upload segment. Jul 28 00:23:37.317387365 (84h47m47.039950405s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317398867 (84h47m47.039961907s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317741793 (84h47m47.040304833s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.317771279 (84h47m47.040334329s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318039613 (84h47m47.040602653s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318294580 (84h47m47.04085762s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318532536 (84h47m47.041095566s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318775812 (84h47m47.041338852s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.318964052 (84h47m47.041527092s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319152892 (84h47m47.041715932s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319350842 (84h47m47.041913882s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319540073 (84h47m47.042103113s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319723063 (84h47m47.042286094s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.319905383 (84h47m47.042468423s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320084705 (84h47m47.042647745s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:37.320267355 (84h47m47.042830395s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Uploading piece... Jul 28 00:23:39.064964023 (84h47m48.787527053s) [upload=81374]: Next returned nil segment Jul 28 00:23:39.064981687 (84h47m48.787544727s) [upload=81374]: Waiting for error group managing segments... Jul 28 00:23:39.340867483 (84h47m49.063430523s) [noise=true , address=45.91.116.9:4262 , piece_id=5BTKLW2XHXG3HN4IFH3MHPY5XHSCLKDORIKXH65S54WGRI7PD4PA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.356965262 (84h47m49.079528302s) [noise=true , address=77.90.31.14:4661 , piece_id=GKA7AOVFST4BGBWTDZY46FRCLXBE5ZMC4VL4TB3ASKGN22CW7SKQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.362359653 (84h47m49.084922693s) [noise=true , address=45.91.116.11:4119 , piece_id=IIJTUIN2CP5XLDSNLG4P25TVIXBOSI7Q6DB5APZK2POKRCLH3FSQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.373141581 (84h47m49.095704651s) [noise=true , address=45.91.116.13:4153 , piece_id=HHYOKEVBOGQZRL5OREZNC4TAR7ZIESB3PQ6BP7QJCLRG4W3Y7ZXQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.430640531 (84h47m49.153203571s) [noise=true , address=77.90.31.11:4553 , piece_id=O3AARDLBWNEJWRLPQGBE6MWIX7DDNXL4THJFVVVL32B6XEBWPEMQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.445133640 (84h47m49.16769667s) [noise=true , address=45.91.116.8:4048 , piece_id=4G5QMO3WCFBOPHCRSBS32EBOFIFZYKZR2KRCBBAYEDLEMEYGB5CA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.454045370 (84h47m49.1766084s) [noise=true , address=77.90.31.12:4589 , piece_id=VNYRWCVDMCODV26ADNGVRAB52D76ITT4JINBZ7WIHQZDYFGNOJDA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.503182252 (84h47m49.225745292s) [noise=true , address=45.91.116.8:4045 , piece_id=FAUR7RYQEGFKKZWHWVSMBBJZ6LDJEY2VFEZZRW577EOF74JVLTRQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.519937578 (84h47m49.242500618s) [noise=true , address=45.91.116.14:4189 , piece_id=7TVR3FJ2VSVLJDBTE3YJM3RIWH4KQULWPVV7GR4UU7PRYUN3UYNA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.522381895 (84h47m49.244944935s) [noise=true , address=45.91.116.5:4299 , piece_id=OPL2HXUU4F47NPFKIBQTJOH5GFDMBKAOGXXAQYIXTLZGH5MRTGAA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.531306760 (84h47m49.2538698s) [noise=true , address=77.90.31.15:4699 , piece_id=5EPMG4C77ZIQXCERZ5X4YJOZ4TH2JECKZLB3GEPRDVRXZA5C2QSA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:23:39.602626742 (84h47m49.325189792s) [noise=true , address=45.91.116.4:4937 , piece_id=323IB6IOPHPPIF2CV6VQKX2JCD4WODZSYEICIL4ASFEQZT3TJBJQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: <nil> Jul 28 00:24:06.434846113 (84h48m16.157409153s) [noise=true , address=104.36.82.7:4912 , piece_id=YPL5PMCIV2XXJGM4VAY4JGHMDCEBGNUL6LT7OUZCJ4IBEESL64XA, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:45336->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434872944 (84h48m16.157435984s) [noise=true , address=104.36.82.9:4006 , piece_id=32TYE6BPAYDHLDFAETEZ7X3RGJOFXZXXEVZAJT2OO4D7G5GKFZVQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434917430 (84h48m16.15748047s) [noise=true , address=104.36.82.15:4258 , piece_id=FW735NPJWY2L2H54LGWOHFZZJKK3I6JDXFFT5B7HNPMJCAMYMGTQ, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:60368->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434930544 (84h48m16.157493584s) [noise=true , address=104.36.82.10:4079 , piece_id=OPWJODZCOT7X6OULN4Y47OVK6MWJQLG2OWYAEZ56WV35MW5MFD3A, seg_pos={0 0}, upload=81374]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:55716->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:06.434955502 (84h48m16.157518542s) [seg_pos={0 0}, upload=81374]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:24:06.663297936 (84h48m16.385860976s) [upload=81374]: Done uploading segments... Jul 28 00:24:06.663325118 (84h48m16.385888158s) [upload=81374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.230925781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X.......X.XX....] duration: 5.834637768s FAIL (2025-07-28 00:23:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.834663508s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.........................................X...XXXXXXXXXXXXXXX] duration: 1m0.233404481s FAIL (2025-07-28 00:23:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== l 28 00:22:59.292329253 (84h47m9.014892293s) [noise=true , address=104.36.82.53:4149 , piece_id=CR2VMGB5R26WM4F4TAV2ZYUSB4VOCEDWFTZQH7CJ3MMZ3JUSGVNA, seg_pos={0 0}, upload=81368]: Uploading piece... Jul 28 00:22:59.292582518 (84h47m9.015145558s) [noise=true , address=104.36.82.21:4490 , piece_id=OD44PHIRHEMAKQD3SHZDROEJRV36WYUSBLO4QUMGASXLG45XYB6A, seg_pos={0 0}, upload=81368]: Uploading piece... Jul 28 00:22:59.292762773 (84h47m9.015325823s) [noise=true , address=104.36.82.38:4701 , piece_id=SIHWUQFBTV5YKLUGTLVOE5KR4JORVB6XRHGWX47CGUK5AVTRW7HQ, seg_pos={0 0}, upload=81368]: Uploading piece... Jul 28 00:22:59.293109396 (84h47m9.015672446s) [noise=true , address=104.36.82.10:4083 , piece_id=KB53CMH5FBIMGGPKAUKKBIOQSM247JH35OUVECYSUWIQKYUJJ3KA, seg_pos={0 0}, upload=81368]: Uploading piece... Jul 28 00:22:59.293231870 (84h47m9.01579491s) [noise=true , address=104.36.82.14:4239 , piece_id=BAKZFJUDCOPK3ZZ53CO7J4A5KD7GGEY2WOQXUPTKVL6ZXJEWZN7Q, seg_pos={0 0}, upload=81368]: Uploading piece... Jul 28 00:23:00.440513029 (84h47m10.163076069s) [upload=81368]: Next returned nil segment Jul 28 00:23:00.440530333 (84h47m10.163093373s) [upload=81368]: Waiting for error group managing segments... Jul 28 00:23:00.758980525 (84h47m10.481543565s) [noise=true , address=64.49.13.137:4498 , piece_id=OZCWNCQWFTHGL65RM7G7ZSBSXX3CKA2QOWGICABNTSCWYIN722WA, seg_pos={0 0}, upload=81368]: Done uploading piece. err: <nil> Jul 28 00:23:58.201030506 (84h48m7.923593546s) [noise=true , address=104.36.82.56:4245 , piece_id=3Z5J7YYBXPOGSJPQQXNGWO3OKE6I2KLSXMRD26GVR3U4D4G5U4GA, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:12YnxyH9TZVuVheQpH9NVRfFUeWHJEtV97wyEJafhCzGZSv13Fn, address:104.36.82.56:4245): protocol: write tcp 188.166.225.170:42494->104.36.82.56:4245: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.201089418 (84h48m7.923652448s) [noise=true , address=104.36.82.54:4170 , piece_id=PKJEJI73QIZX2M7M3JH5MQD5W5UNL54N2YR3FFO7N3EBO5SI3IZQ, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX, address:104.36.82.54:4170): protocol: write tcp 188.166.225.170:49314->104.36.82.54:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.201188899 (84h48m7.923751939s) [noise=true , address=104.36.82.53:4149 , piece_id=CR2VMGB5R26WM4F4TAV2ZYUSB4VOCEDWFTZQH7CJ3MMZ3JUSGVNA, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: write tcp 188.166.225.170:38382->104.36.82.53:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.201256237 (84h48m7.923819277s) [noise=true , address=104.36.82.41:4792 , piece_id=VFBXO36AR5GJZ5SAENEUYELFKEN3SWLFUTWOWON4MXIVB3W5TCOA, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: write tcp 188.166.225.170:37654->104.36.82.41:4792: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.201308137 (84h48m7.923871177s) [noise=true , address=104.36.82.56:4233 , piece_id=DGS5ELIPA6FEA7YAT2CNBKWI7X36JODKGVTYFDTTPPVKEPOJOXTA, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: write tcp 188.166.225.170:56816->104.36.82.56:4233: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.201340458 (84h48m7.923903498s) [noise=true , address=104.36.82.52:4114 , piece_id=MHLB77O27LF4BSZLYXM25NVGUV4CQGX7V2NHE377SAAVKLZM3IVA, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:12PXReFMGqAhCAp5BSE3vsJdfRAUathsJLdB8Guz7DzW6UHmATM, address:104.36.82.52:4114): protocol: write tcp 188.166.225.170:47784->104.36.82.52:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.201417847 (84h48m7.923980887s) [noise=true , address=104.36.82.10:4083 , piece_id=KB53CMH5FBIMGGPKAUKKBIOQSM247JH35OUVECYSUWIQKYUJJ3KA, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh, address:104.36.82.10:4083): protocol: write tcp 188.166.225.170:42964->104.36.82.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.201486418 (84h48m7.924049499s) [noise=true , address=104.36.82.14:4239 , piece_id=BAKZFJUDCOPK3ZZ53CO7J4A5KD7GGEY2WOQXUPTKVL6ZXJEWZN7Q, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:104.36.82.14:4239): protocol: write tcp 188.166.225.170:46790->104.36.82.14:4239: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.201739633 (84h48m7.924302673s) [noise=true , address=104.36.82.47:4537 , piece_id=J3YXEKRZ5T5DUUVNN7SC4HVRIBTIIUKPMF4REPWI4KJWZZPU4YFA, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: write tcp 188.166.225.170:39476->104.36.82.47:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.201993368 (84h48m7.924556408s) [noise=true , address=104.36.82.15:4262 , piece_id=3GDFXDBYCA7IHG52FH2HPJZRKKVPDPALCBMIYE2NQVCOJELTA5YQ, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, address:104.36.82.15:4262): protocol: write tcp 188.166.225.170:60256->104.36.82.15:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.202060126 (84h48m7.924623166s) [noise=true , address=104.36.82.42:4830 , piece_id=JKKQFSGLACEFWQQ5RSKQCBTFGFKWWBC2HQW6YCHEKKKHXMHYEAHQ, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: write tcp 188.166.225.170:44922->104.36.82.42:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.202190455 (84h48m7.924753495s) [noise=true , address=104.36.82.20:4447 , piece_id=X3MJOYE7BMU2R6FDINZQ67BGGDKJWDCW4JAK6RU5T5MPEF3QFUXQ, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:1Li8ZwrWGPx5Y478pYLwnWJ7N2x3dbXgwGaUGWVtVhvv61EnTs, address:104.36.82.20:4447): protocol: write tcp 188.166.225.170:38358->104.36.82.20:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.202227556 (84h48m7.924790597s) [noise=true , address=104.36.82.22:4514 , piece_id=747O5XDAVCQV4AR2QIFWRB3AFDX3WNVMG55HP4LVUS7LO3ZWLEOQ, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: write tcp 188.166.225.170:58514->104.36.82.22:4514: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.202243146 (84h48m7.924806186s) [noise=true , address=104.36.82.21:4490 , piece_id=OD44PHIRHEMAKQD3SHZDROEJRV36WYUSBLO4QUMGASXLG45XYB6A, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n, address:104.36.82.21:4490): protocol: write tcp 188.166.225.170:46764->104.36.82.21:4490: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.202267112 (84h48m7.924830152s) [noise=true , address=104.36.82.38:4701 , piece_id=SIHWUQFBTV5YKLUGTLVOE5KR4JORVB6XRHGWX47CGUK5AVTRW7HQ, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:104.36.82.38:4701): protocol: write tcp 188.166.225.170:51160->104.36.82.38:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.202277331 (84h48m7.924840371s) [noise=true , address=104.36.82.16:4313 , piece_id=666KHT2IY2AKP4MXYM6DGFXD4HJTG3JV2MVEWGUKHLJXVMY5Q2GA, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:104.36.82.16:4313): protocol: write tcp 188.166.225.170:45428->104.36.82.16:4313: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.202323089 (84h48m7.924886129s) [noise=true , address=104.36.82.11:4143 , piece_id=SJXRDNJAF5SXLV3226GIVKNZNDLWP2AEE3Q33OXAVFUGMPIF5QJQ, seg_pos={0 0}, upload=81368]: Done uploading piece. err: ecclient: upload failed (node:12ipThd6PwZUpCXy3ML4qqMd9dmRLmxKjVNCNoT5wCg1szdWp8U, address:104.36.82.11:4143): protocol: write tcp 188.166.225.170:51106->104.36.82.11:4143: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:58.202348778 (84h48m7.924911818s) [seg_pos={0 0}, upload=81368]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:23:58.432373961 (84h48m8.154937052s) [upload=81368]: Done uploading segments... Jul 28 00:23:58.432407465 (84h48m8.154970505s) [upload=81368]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.233427916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................X............X..................] duration: 1.958783928s FAIL (2025-07-28 00:23:49 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.958813194s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................X............X.................] duration: 1.825227552s FAIL (2025-07-28 00:23:31 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.825253884s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X.......X.XX...] duration: 6.060532211s FAIL (2025-07-28 00:23:01 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 6.06055843s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..........................................X...XXXXXXXXXXXXXX] duration: 1m0.23325698s FAIL (2025-07-28 00:22:57 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== PLIDCDPPKPAWBSDUBQ, seg_pos={0 0}, upload=81354]: Uploading piece... Jul 28 00:21:58.886454683 (84h46m8.609017723s) [noise=true , address=104.36.82.10:4081 , piece_id=FOHRNMCII4RCHPZ6YCY6MCSNTHWEQCRJUFGGIJDFRGLO3A5HRXJQ, seg_pos={0 0}, upload=81354]: Uploading piece... Jul 28 00:21:58.886679262 (84h46m8.609242302s) [noise=true , address=104.36.82.52:4093 , piece_id=Y5Q3NMWT6MXYWJ2MLZSA65HSWQIMOZU6UJLPTACJH6CMOABP7A3Q, seg_pos={0 0}, upload=81354]: Uploading piece... Jul 28 00:21:58.886869927 (84h46m8.609432957s) [noise=true , address=104.36.82.21:4475 , piece_id=TVSPNVVH776IULW3WWO7EC4EKBXEEBCSUOMJV3BWIAXJFXAZMFJA, seg_pos={0 0}, upload=81354]: Uploading piece... Jul 28 00:21:58.887141717 (84h46m8.609704757s) [noise=true , address=104.36.82.42:4828 , piece_id=ONM3LZCQGCCXGXWHDDVCD4HCJRPRKJKYX35FQ3PAFGFWXPXIAM6Q, seg_pos={0 0}, upload=81354]: Uploading piece... Jul 28 00:21:58.887337111 (84h46m8.609900151s) [noise=true , address=104.36.82.4:4808 , piece_id=LLZZ5E4Y36BUUG6MN6CKZS7TM5IBJ2Z3MNR4RLTDOECYOUH45RAQ, seg_pos={0 0}, upload=81354]: Uploading piece... Jul 28 00:21:58.887538707 (84h46m8.610101757s) [noise=true , address=104.36.82.8:4002 , piece_id=JKSQOVLJOJEFGKS6AWG5YES7CIPRKAN4C7IIFH2VAUB35V7WM55Q, seg_pos={0 0}, upload=81354]: Uploading piece... Jul 28 00:22:57.744275231 (84h47m7.466838271s) [upload=81354]: Next segment err: context deadline exceeded Jul 28 00:22:57.745079249 (84h47m7.467642289s) [noise=true , address=104.36.82.55:4227 , piece_id=5OWQMIJN33DO2L5AZLZEHBE5DJGFNRPUNQVATY7D622CWEQOQZWQ, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:19qGGwWT9xUYsJyRnFHyHKPFr6g4AsckxRNVTg2shfVhoQtcCz, address:104.36.82.55:4227): protocol: write tcp 188.166.225.170:33766->104.36.82.55:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745123514 (84h47m7.467686544s) [noise=true , address=104.36.82.12:4160 , piece_id=4PMBUQ5MSXWOW2S3OB3KCLA5RJKNJAZ73XZPKH66POWVFQQ7G4UA, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:12dggZZRv5coYBNJEBvHNqWtMSqYkZoXYmitc9nBKJZftnsX17D, address:104.36.82.12:4160): protocol: rpc: dial tcp 104.36.82.12:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745159913 (84h47m7.467722953s) [noise=true , address=104.36.82.47:4528 , piece_id=EB2AUW7ZBLEJGPY2IBHDTYF3FH2N277HYNQEVN6WIOUZXWVIJDLQ, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: rpc: dial tcp 104.36.82.47:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745193958 (84h47m7.467756998s) [noise=true , address=104.36.82.3:4800 , piece_id=STG5632YZNEE46FBCS26H7KKETZUWWRUHTXWJOAQDFFIFW2LU4AA, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:12CWPcNRnN5pX4mAU6Zkbwgy2teaAfusbmeH7niPaY9r2yiDLtT, address:104.36.82.3:4800): protocol: rpc: dial tcp 104.36.82.3:4800: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745224005 (84h47m7.467787036s) [noise=true , address=104.36.82.42:4828 , piece_id=ONM3LZCQGCCXGXWHDDVCD4HCJRPRKJKYX35FQ3PAFGFWXPXIAM6Q, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:1cX7FtKC8KMopnX9zvkT6fYdnutK84cFjLW6xMwpXuGsqpiN8k, address:104.36.82.42:4828): protocol: write tcp 188.166.225.170:55156->104.36.82.42:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745245687 (84h47m7.467808727s) [noise=true , address=104.36.82.15:4272 , piece_id=55ZA47SKNOGSI2BHTO624RXRF7LHHZ26HRYT6IRS4QNPRZPNMW4A, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi, address:104.36.82.15:4272): protocol: write tcp 188.166.225.170:37662->104.36.82.15:4272: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745263652 (84h47m7.467826682s) [noise=true , address=104.36.82.50:4036 , piece_id=NZYHQIYLFQFQKVV3AKIFQRI5KIGZ7SCDIBV5LDXFW4GU7AOLLBHQ, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:17cebKoBWpDfVVnHqUDtFfJ71Lz8B7DqFonLumECeHxHf5o4sY, address:104.36.82.50:4036): protocol: write tcp 188.166.225.170:49034->104.36.82.50:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745292086 (84h47m7.467855126s) [noise=true , address=104.36.82.41:4769 , piece_id=Q62RBHGWHWLCGQ4K6N2XQGV6356ZFJI37DKDM3V77VFJITEPHCTA, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:12EhEYh6rUckEz9JaSFuj52LUM3ujuYaGif4K4jYrE4r5PMsBsj, address:104.36.82.41:4769): protocol: write tcp 188.166.225.170:34702->104.36.82.41:4769: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745303488 (84h47m7.467866528s) [noise=true , address=104.36.82.10:4081 , piece_id=FOHRNMCII4RCHPZ6YCY6MCSNTHWEQCRJUFGGIJDFRGLO3A5HRXJQ, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:12t9rGfCQ7bLkNnZzaPVgkE2Y9rXsiFp1XGdeECzv9M1jk8Qqmp, address:104.36.82.10:4081): protocol: write tcp 188.166.225.170:41360->104.36.82.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745330810 (84h47m7.46789385s) [noise=true , address=104.36.82.54:4187 , piece_id=DBJJFGRZ6TEXNXGDQNMS3RUXJHASXIWGMZNBDXGVAH3SYOTPCD2Q, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF, address:104.36.82.54:4187): protocol: write tcp 188.166.225.170:40450->104.36.82.54:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745370987 (84h47m7.467934017s) [noise=true , address=104.36.82.21:4475 , piece_id=TVSPNVVH776IULW3WWO7EC4EKBXEEBCSUOMJV3BWIAXJFXAZMFJA, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:104.36.82.21:4475): protocol: write tcp 188.166.225.170:54294->104.36.82.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745404421 (84h47m7.467967461s) [noise=true , address=104.36.82.5:4853 , piece_id=5RXUMEQASDXXAAL7WLDW7XHEF3WXMTOSXKPLIDCDPPKPAWBSDUBQ, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 188.166.225.170:55536->104.36.82.5:4853: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745424199 (84h47m7.467987239s) [noise=true , address=104.36.82.13:4213 , piece_id=R2E2LGE5GEQU6JG4VO2QNQOWNQH7JMMZWF4GZDRGQG5LCMQVKNFQ, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:12XDcAySWNCL7XoStP5GKzAksN1kTs9L8u4KStgyaiQn3AvLcRM, address:104.36.82.13:4213): protocol: write tcp 188.166.225.170:41476->104.36.82.13:4213: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745439558 (84h47m7.468002598s) [noise=true , address=104.36.82.52:4093 , piece_id=Y5Q3NMWT6MXYWJ2MLZSA65HSWQIMOZU6UJLPTACJH6CMOABP7A3Q, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: write tcp 188.166.225.170:55160->104.36.82.52:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745465097 (84h47m7.468028137s) [noise=true , address=104.36.82.49:4008 , piece_id=FJZ7LTHT7DB5HZBHELLSG244SYO2VR7KOSTPZ3YLXFUFNK4OWQHA, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:12KYDzfd4KtZm1cyRXCKyntNLnoLYCBdPubsebpdi8H6khSpeAN, address:104.36.82.49:4008): protocol: rpc: dial tcp 104.36.82.49:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745472912 (84h47m7.468035952s) [noise=true , address=104.36.82.4:4808 , piece_id=LLZZ5E4Y36BUUG6MN6CKZS7TM5IBJ2Z3MNR4RLTDOECYOUH45RAQ, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:1fbQCaGY9ZssBzyNusnhyY994e8ytvW4sQDQRt7q2tdGrg6ps3, address:104.36.82.4:4808): protocol: rpc: dial tcp 104.36.82.4:4808: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745478363 (84h47m7.468041403s) [noise=true , address=104.36.82.53:4138 , piece_id=HWZWEDHOHNE2WPW7RATLCUAP7HP4GFZBRMBD6BBL4P7WAC7UACPA, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: write tcp 188.166.225.170:40304->104.36.82.53:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745487320 (84h47m7.46805036s) [noise=true , address=104.36.82.8:4002 , piece_id=JKSQOVLJOJEFGKS6AWG5YES7CIPRKAN4C7IIFH2VAUB35V7WM55Q, seg_pos={0 0}, upload=81354]: Done uploading piece. err: ecclient: upload failed (node:1DW1BZ57L7XPH8kTjCqKxBc492jeqwb7eKni3zYjVetMBrmyJ9, address:104.36.82.8:4002): protocol: write tcp 188.166.225.170:33566->104.36.82.8:4002: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:57.745512017 (84h47m7.468075057s) [seg_pos={0 0}, upload=81354]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.233273242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 5.83407122s FAIL (2025-07-28 00:22:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.834100315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................X............X................] duration: 1.860885024s FAIL (2025-07-28 00:22:43 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.860907075s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X.......X.XX..] duration: 5.858184883s FAIL (2025-07-28 00:22:35 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.858207606s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...........................................X...XXXXXXXXXXXXX] duration: 1m0.235410925s FAIL (2025-07-28 00:21:57 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== CM4P7CSST75SNL56XJ7Q, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:20:58.394239288 (84h45m8.116802328s) [noise=true , address=104.36.82.6:4879 , piece_id=RUYOAVTD77CHSLKSE7ZOQFBJJ4QBDJ3GWNCQTZDKSCXH4HXLD3DQ, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:20:58.394442326 (84h45m8.117005367s) [noise=true , address=104.36.82.55:4202 , piece_id=WJE62QI7YVUGLWV7UMRNUDH5GZPZY6O44MKRWUBZV2KBYHE7M4MA, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:20:58.394642330 (84h45m8.11720537s) [noise=true , address=104.36.82.52:4104 , piece_id=GIDJCXWDKL33QIQNYMV4RSXXERZNNX5ZM2STNNT74RBKLOAY5GKA, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:20:58.394852391 (84h45m8.117415431s) [noise=true , address=104.36.82.9:4014 , piece_id=WPGIYXAVRE5LNAC7BTIMYM2ZLSFK4I7EKBFNUZHXCQXFW2CKIK7Q, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:20:58.395072282 (84h45m8.117635322s) [noise=true , address=104.36.82.53:4130 , piece_id=JFZZSYH52WQ6EBH6ZPKR7XPU22PXD3FMPX4TQPKU3YN77FL3SHTA, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:21:57.287344145 (84h46m7.009907195s) [upload=81337]: Next segment err: context deadline exceeded Jul 28 00:21:57.287832549 (84h46m7.01039559s) [noise=true , address=104.36.82.49:4985 , piece_id=QJYTPFDOMOQ5SQCQV5XJEQ4SHRYAUV64WSVFLH5JCLRMLOMC2LUA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: write tcp 188.166.225.170:55326->104.36.82.49:4985: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.287873427 (84h46m7.010436467s) [noise=true , address=104.36.82.46:4498 , piece_id=XTAL5NZOVEUKKQ5AEVVFCZNGB3T5DSQOEP2ZFZ3C464PE2BWRNMQ, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:1jyjxdWkBRBkDHSKknNmPdcnJ3HRcSCUQNnmpb1CaiEdMPEGk4, address:104.36.82.46:4498): protocol: write tcp 188.166.225.170:54314->104.36.82.46:4498: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.287907683 (84h46m7.010470724s) [noise=true , address=104.36.82.6:4879 , piece_id=RUYOAVTD77CHSLKSE7ZOQFBJJ4QBDJ3GWNCQTZDKSCXH4HXLD3DQ, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12jH7W99nQ77T67b8WVVqmg9Jqi5QoB1qM5Wcj8WexywAZ7fRNT, address:104.36.82.6:4879): protocol: write tcp 188.166.225.170:46190->104.36.82.6:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.287945856 (84h46m7.010508906s) [noise=true , address=104.36.82.53:4130 , piece_id=JFZZSYH52WQ6EBH6ZPKR7XPU22PXD3FMPX4TQPKU3YN77FL3SHTA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12N8MGboFyEV381rkrBva3G3advz3QQEY2HCHxwqzKX2X5zxNvb, address:104.36.82.53:4130): protocol: write tcp 188.166.225.170:48636->104.36.82.53:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288059634 (84h46m7.010622674s) [noise=true , address=104.36.82.42:4798 , piece_id=RXFPR7QOYXHUKDUDSGKDVYG2YR4HV7XZ2PGDNFJGWHG6C3RPZLWQ, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:51402->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288121732 (84h46m7.010684772s) [noise=true , address=104.36.82.50:4043 , piece_id=BZYCXUKWSWJV47EZRGTXFZYNMKPSBMHAYTKW2LC767M7H6NKU7YQ, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:104.36.82.50:4043): protocol: write tcp 188.166.225.170:58482->104.36.82.50:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288159245 (84h46m7.010722285s) [noise=true , address=104.36.82.36:4632 , piece_id=U276W6EIX52I3CG3BJ3VV7ANMPUYIFBQNIOCEUT4XR7IG3UMW2TA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: write tcp 188.166.225.170:40544->104.36.82.36:4632: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288203669 (84h46m7.010766709s) [noise=true , address=104.36.82.10:4090 , piece_id=RRPNRXMQGBFWVSL6IJP6C5EHDUHQVUUHCM4P7CSST75SNL56XJ7Q, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:104.36.82.10:4090): protocol: write tcp 188.166.225.170:60812->104.36.82.10:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288239388 (84h46m7.010802428s) [noise=true , address=104.36.82.52:4104 , piece_id=GIDJCXWDKL33QIQNYMV4RSXXERZNNX5ZM2STNNT74RBKLOAY5GKA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12qmPpCmXAvuJZn2KR5Kr4nHYwKEzWvwqZ82nqNka5u1mKaZmXs, address:104.36.82.52:4104): protocol: write tcp 188.166.225.170:38968->104.36.82.52:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288277020 (84h46m7.01084006s) [noise=true , address=104.36.82.54:4175 , piece_id=2WV4ISHIQZ3NQXIUG5PRMWXV2NGUZ5Q5VKGZQ2KUIPT7YI2OOSHA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: write tcp 188.166.225.170:57302->104.36.82.54:4175: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288441444 (84h46m7.011004484s) [noise=true , address=104.36.82.14:4242 , piece_id=RXXG7CX2O4C4333X7EHQOUHR4ZGPLBA7K66PJYOEMO5W4W6IZVWQ, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz, address:104.36.82.14:4242): protocol: write tcp 188.166.225.170:60670->104.36.82.14:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288512010 (84h46m7.01107506s) [noise=true , address=104.36.82.56:4258 , piece_id=WGLDNQYGXJKCDXYSRM72H7GECMRHECFFEPZ25IJQLMWJQGTBSE4A, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12g7DdMjQHyYTD9hRvixfNedxyPr6q6uE2rLdEbhQ1MGANRMDm3, address:104.36.82.56:4258): protocol: write tcp 188.166.225.170:35042->104.36.82.56:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288578076 (84h46m7.011141156s) [noise=true , address=104.36.82.12:4174 , piece_id=2QAW6BIB4I74IKMBZEDN5A2PV7GZBMKPL4IMYWDZUGBWVWFY454A, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12ZqPqvh4SVpDmkpVKQXZUW48CtzV9VMeMGJoDZwA7KpNfmjfow, address:104.36.82.12:4174): protocol: write tcp 188.166.225.170:37372->104.36.82.12:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288669540 (84h46m7.011232611s) [noise=true , address=104.36.82.9:4014 , piece_id=WPGIYXAVRE5LNAC7BTIMYM2ZLSFK4I7EKBFNUZHXCQXFW2CKIK7Q, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12uK9ULUdABDm5XBP4nh1omWhR5qoQerFsCmL4bQ9dmNespXpTy, address:104.36.82.9:4014): protocol: write tcp 188.166.225.170:32806->104.36.82.9:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288696572 (84h46m7.011259612s) [noise=true , address=104.36.82.7:4942 , piece_id=HGXN6AR6HVLTCDTRO6Q7UBSZDQ7JY36M52U7OK5M5RR5TH43EN3Q, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:4942): protocol: write tcp 188.166.225.170:52758->104.36.82.7:4942: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288748462 (84h46m7.011311502s) [noise=true , address=104.36.82.55:4202 , piece_id=WJE62QI7YVUGLWV7UMRNUDH5GZPZY6O44MKRWUBZV2KBYHE7M4MA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: write tcp 188.166.225.170:58294->104.36.82.55:4202: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288755485 (84h46m7.011318525s) [noise=true , address=104.36.82.15:4280 , piece_id=XSWP44ZZGA2JCQA6LQVCM3TIYNFGKJE7X427Q7ABVKSBZUKRT2EA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:1t4hMpMGCu1YMmNLX6fssTjWy9Xi5Fp8kdHTDmkhxH9qsZiYM6, address:104.36.82.15:4280): protocol: write tcp 188.166.225.170:60296->104.36.82.15:4280: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288784791 (84h46m7.011347831s) [noise=true , address=104.36.82.11:4125 , piece_id=GHLIH2KBA6WEA53ZXLJ3NTQRGY46OHN4R5ZG365VAIKPRZMRHKZA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:4125): protocol: write tcp 188.166.225.170:43330->104.36.82.11:4125: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288825339 (84h46m7.011388379s) [seg_pos={0 0}, upload=81337]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.235422326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................X............X...............] duration: 1.920898625s FAIL (2025-07-28 00:21:49 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.920954442s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 5.841853422s FAIL (2025-07-28 00:21:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.841882569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................X.......X.XX.] duration: 5.820141665s FAIL (2025-07-28 00:21:32 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.820169569s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................X............X..............] duration: 1.845903813s FAIL (2025-07-28 00:21:31 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.845932357s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 5.824791367s FAIL (2025-07-28 00:21:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.824824901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................X............X.............] duration: 2.133415321s FAIL (2025-07-28 00:21:14 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.133444076s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [............................................X...XXXXXXXXXXXX] duration: 1m0.239743358s FAIL (2025-07-28 00:20:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== Jul 28 00:19:57.906994140 (84h44m7.62955718s) [noise=true , address=104.36.82.11:4137 , piece_id=YQC4N2XEZZR6Q4XXMS3NUW7GLHF2JVQP6U2IH3XPEFPNJD23AMVQ, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907128808 (84h44m7.629691838s) [noise=true , address=104.36.82.4:4831 , piece_id=RGGSZZVZXNG4AYAKAVJ4NCZF233LYIIEQIQZ77RHMUUJ67XF6CFA, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907285447 (84h44m7.629848477s) [noise=true , address=104.36.82.14:4253 , piece_id=6D5W2GS5UI4EN5ODH446BZILZJIE5NFDVAHBGXVLJR2DOUJ3D4DA, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907359248 (84h44m7.629922278s) [noise=true , address=104.36.82.53:4150 , piece_id=NH2SET23UMC75UWQ7LN3O4JCQYEENWEE7UXLJLPFOZWVCXJEXQZQ, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907579319 (84h44m7.630142359s) [noise=true , address=104.36.82.6:4897 , piece_id=ZDQBPT3KIMDFS44JL3YOJVVKYRXIK56IGBGJ35OPJGPYVXJVIU3A, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907681365 (84h44m7.630244405s) [noise=true , address=79.127.201.215:4360 , piece_id=YXCCPQ3TFYRMJFL3ESWPEG73WNEM642BHBMBUJY47DU7WT2VXJTQ, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907895695 (84h44m7.630458735s) [noise=true , address=104.36.82.54:4173 , piece_id=LKPF6JIDWQOYZLMZURODZLFTNTODGHDC2KPKKRYPACWEDBYL6U7Q, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.908148118 (84h44m7.630711148s) [noise=true , address=104.36.82.49:4002 , piece_id=WUCF4NTRVRJDJQU6AXHEMSVWKJN4N7SP5EL6NLD7P3U6OEXMESRA, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:59.417184952 (84h44m9.139747992s) [upload=81322]: Next returned nil segment Jul 28 00:19:59.417201624 (84h44m9.139764664s) [upload=81322]: Waiting for error group managing segments... Jul 28 00:20:01.618731737 (84h44m11.341294777s) [noise=true , address=79.127.201.215:4360 , piece_id=YXCCPQ3TFYRMJFL3ESWPEG73WNEM642BHBMBUJY47DU7WT2VXJTQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: <nil> Jul 28 00:20:56.806154697 (84h45m6.528717747s) [noise=true , address=104.36.82.11:4137 , piece_id=YQC4N2XEZZR6Q4XXMS3NUW7GLHF2JVQP6U2IH3XPEFPNJD23AMVQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:104.36.82.11:4137): protocol: rpc: dial tcp 104.36.82.11:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806261239 (84h45m6.52882428s) [noise=true , address=104.36.82.5:4847 , piece_id=7DX6JDL2RUQONZBBVBAGINWXM3Y45ZPZNYKP3YDCFDDH63ME4BUA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:4847): protocol: rpc: dial tcp 104.36.82.5:4847: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806371210 (84h45m6.52893426s) [noise=true , address=104.36.82.41:4771 , piece_id=ERGGVKVMJYT2FOLVYIBJLMNTBOAWKGEP7T7NK67S2JO3DEYDTRPA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto, address:104.36.82.41:4771): protocol: rpc: dial tcp 104.36.82.41:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806428981 (84h45m6.528992021s) [noise=true , address=104.36.82.9:4012 , piece_id=HR5NRTEQOVO2J446PWCX5XTTPCRO4Z5PRXDATEF5PZYTXBEE5QPQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:15gHkND99Ksp8TdyAbAvMUymRoLDyqvbdcy9Y3Hjhpb8zTpCHi, address:104.36.82.9:4012): protocol: rpc: dial tcp 104.36.82.9:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806463738 (84h45m6.529026778s) [noise=true , address=104.36.82.46:4508 , piece_id=HRNWSQC5OUUWKF2IE5EVGEPMVWDJSTGSW6AC4MZYSD3RUPTK6SAQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: rpc: dial tcp 104.36.82.46:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806502431 (84h45m6.529065471s) [noise=true , address=104.36.82.51:4053 , piece_id=4MA3SWYEIZGJCVAXRPYV2ZMF3RGVJPPCXD2WBRILJZBKJ7IW3QZA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG, address:104.36.82.51:4053): protocol: rpc: dial tcp 104.36.82.51:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806538690 (84h45m6.52910173s) [noise=true , address=104.36.82.4:4831 , piece_id=RGGSZZVZXNG4AYAKAVJ4NCZF233LYIIEQIQZ77RHMUUJ67XF6CFA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806573928 (84h45m6.529136968s) [noise=true , address=104.36.82.14:4253 , piece_id=6D5W2GS5UI4EN5ODH446BZILZJIE5NFDVAHBGXVLJR2DOUJ3D4DA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14, address:104.36.82.14:4253): protocol: rpc: dial tcp 104.36.82.14:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806611400 (84h45m6.52917444s) [noise=true , address=104.36.82.20:4456 , piece_id=72LVGO6GBBUH3ZUTH5R5ZXDZUV6CGI4YN6MG7DT3D4ZPMDFDQC4A, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:104.36.82.20:4456): protocol: write tcp 188.166.225.170:36146->104.36.82.20:4456: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806809559 (84h45m6.529372599s) [noise=true , address=104.36.82.15:4275 , piece_id=NG3ISLHLF5M2FZYQQVDK6523NBL7CFFQG7KSQIFUE4ADNJL3CULQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12EzSYYm7aKxspeyC4hZCxZzCdcSNMyDTJx3TYcci5bVT6rZLbr, address:104.36.82.15:4275): protocol: rpc: dial tcp 104.36.82.15:4275: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806923376 (84h45m6.529486426s) [noise=true , address=104.36.82.10:4096 , piece_id=ZMMS7B35OUXL6XPEUCZUTI4QT3FOODKW5M7FN33SQS26QAHT6S2A, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA, address:104.36.82.10:4096): protocol: rpc: dial tcp 104.36.82.10:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807009181 (84h45m6.529572221s) [noise=true , address=104.36.82.6:4897 , piece_id=ZDQBPT3KIMDFS44JL3YOJVVKYRXIK56IGBGJ35OPJGPYVXJVIU3A, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12qtpRnAcGiHmdR6J6ysdwaLPvejnCvpKm6MggqsezhbZ14Z5Bt, address:104.36.82.6:4897): protocol: rpc: dial tcp 104.36.82.6:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807015794 (84h45m6.529578834s) [noise=true , address=104.36.82.56:4255 , piece_id=VEVSTLO4GUHKIIRVG64D2U52ANQPHVZ4M2MUKFLQICAIZ2J34UCA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12S6rwimXBvaEb4KneAwL5H3t5zTmvGVhyJSWCTU1u17r4RWJta, address:104.36.82.56:4255): protocol: rpc: dial tcp 104.36.82.56:4255: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807050169 (84h45m6.529613209s) [noise=true , address=104.36.82.49:4002 , piece_id=WUCF4NTRVRJDJQU6AXHEMSVWKJN4N7SP5EL6NLD7P3U6OEXMESRA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo, address:104.36.82.49:4002): protocol: rpc: dial tcp 104.36.82.49:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807088282 (84h45m6.529651322s) [noise=true , address=104.36.82.53:4133 , piece_id=5PDEOLN4QJB4RS6ANJIWOCALNZZQDOAF23QV2ELJFRY6ZS4AJ5YA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: rpc: dial tcp 104.36.82.53:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807101417 (84h45m6.529664478s) [noise=true , address=104.36.82.53:4150 , piece_id=NH2SET23UMC75UWQ7LN3O4JCQYEENWEE7UXLJLPFOZWVCXJEXQZQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: rpc: dial tcp 104.36.82.53:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807131305 (84h45m6.529694345s) [noise=true , address=104.36.82.54:4173 , piece_id=LKPF6JIDWQOYZLMZURODZLFTNTODGHDC2KPKKRYPACWEDBYL6U7Q, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: write tcp 188.166.225.170:44964->104.36.82.54:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807167974 (84h45m6.529731034s) [seg_pos={0 0}, upload=81322]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:20:57.043655985 (84h45m6.766219025s) [upload=81322]: Done uploading segments... Jul 28 00:20:57.043682235 (84h45m6.766245275s) [upload=81322]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.23975456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................X............X............] duration: 1.902202206s FAIL (2025-07-28 00:20:37 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.902225211s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................X.......X.XX] duration: 4.075580564s FAIL (2025-07-28 00:20:18 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 4.075608299s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 5.780575958s FAIL (2025-07-28 00:20:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.780602227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................X............X...........] duration: 1.860872301s FAIL (2025-07-28 00:19:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.860895195s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.............................................X...XXXXXXXXXXX] duration: 1m0.238740178s FAIL (2025-07-28 00:19:56 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== N6YU47IQ2RA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.440410160 (84h43m7.16297321s) [noise=true , address=104.36.82.40:4973 , piece_id=UWOSZID6YLTHO54RBS4ZK3SFK2PA4OVTJB2VYY655KMFTTTAZ4VQ, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.440737477 (84h43m7.163300527s) [noise=true , address=104.36.82.18:4396 , piece_id=RWV6DS3ELZM5IAT2WX22T25UWIYMBRRJAUKMA4QSGN45PO3MRQAA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.441031469 (84h43m7.163594509s) [noise=true , address=104.36.82.56:4246 , piece_id=NHHZS7SADGQTZSWUEM6PUYT7LMXKGSLIAEO4YZDZXXZTENWCVFFA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.441321363 (84h43m7.163884414s) [noise=true , address=104.36.82.16:4301 , piece_id=VWKOS6JM7SSSSGRUYHJDVYQJLT645Z4NJYRF3TU4VXBVHSDID4OA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.441595598 (84h43m7.164158628s) [noise=true , address=104.36.82.55:4202 , piece_id=QAPD6CCJV5V5O5S2O5JYHSD6O5YU7DIKP3NSXL3FY4OPHLDK35XA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.442018017 (84h43m7.164581057s) [noise=true , address=104.36.82.53:4154 , piece_id=OQROWXS6GFWUNVE43TS6F3QMFXLIKSHEARAAYAV2WZJMPMMJJTQQ, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.442098872 (84h43m7.164661912s) [noise=true , address=104.36.82.42:4807 , piece_id=MXI7PPAP6PXKQGO67NESW6ZVWFLF4HRENZKRGXOTA3JF7LTCIQJQ, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.442410307 (84h43m7.164973347s) [noise=true , address=104.36.82.50:4032 , piece_id=DSXLYFSLOZY2PB73IYN2TDJWIXUGXMEPHDUWOGHJ3QR3FWV75UBA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:19:56.344684181 (84h44m6.067247221s) [upload=81307]: Next segment err: context deadline exceeded Jul 28 00:19:56.345169609 (84h44m6.06773265s) [noise=true , address=104.36.82.21:4507 , piece_id=RYD5CA2MZF4ZG5VL4SG5LDKAN2VQ2RSGKXRXOD275HK4IO2V3QOQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1XW3SaTU4fPuKGNeLtt4H9wQXw5ADEcvzcpo7J53wUULsMT5Bv, address:104.36.82.21:4507): protocol: rpc: dial tcp 104.36.82.21:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.345309437 (84h44m6.067872477s) [noise=true , address=104.36.82.55:4202 , piece_id=QAPD6CCJV5V5O5S2O5JYHSD6O5YU7DIKP3NSXL3FY4OPHLDK35XA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: rpc: dial tcp 104.36.82.55:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.345347060 (84h44m6.06791009s) [noise=true , address=104.36.82.42:4807 , piece_id=MXI7PPAP6PXKQGO67NESW6ZVWFLF4HRENZKRGXOTA3JF7LTCIQJQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1iTiX3X69T9rKBxe1MLwpJEJPZyXVATQBP9cDEnQe2riXMGZeZ, address:104.36.82.42:4807): protocol: rpc: dial tcp 104.36.82.42:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.345376866 (84h44m6.067939897s) [noise=true , address=104.36.82.54:4190 , piece_id=ZH5TFAYYBR2NDUFBKRZPQBBIFGXR6VBAIJKMNUYCF5TARZIGAGOA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: rpc: dial tcp 104.36.82.54:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.345408647 (84h44m6.067971687s) [noise=true , address=104.36.82.12:4167 , piece_id=6NCVVGGNFC3TZQAZRQ5XGSND3KKJWE5SJRUDTTF53MOSUYUAXZ3Q, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:59842->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.345436170 (84h44m6.06799921s) [noise=true , address=104.36.82.18:4396 , piece_id=RWV6DS3ELZM5IAT2WX22T25UWIYMBRRJAUKMA4QSGN45PO3MRQAA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1k5gzJBGWhpARcfiynburgfTqJ2t7sVNs1mAkUfToMG5saxPqx, address:104.36.82.18:4396): protocol: rpc: dial tcp 104.36.82.18:4396: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346098667 (84h44m6.068661707s) [noise=true , address=104.36.82.52:4112 , piece_id=BDBGZ5LXAWLK6BKOXQDLH3QVGP3MGGSYAHKGZEBUVJGXPL5ZARRQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1dQFyMxQ1vVeHWCmyUWFUtJNBTdRUP2bbdLJQCMwF9z1L1eapD, address:104.36.82.52:4112): protocol: rpc: dial tcp 104.36.82.52:4112: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346133043 (84h44m6.068696083s) [noise=true , address=104.36.82.9:4009 , piece_id=QHGJVTXXRGOBNYKRJ2WXXAZ3MSGTLX2WDGEAUC3GMPSMHMUMHUIQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346161317 (84h44m6.068724357s) [noise=true , address=104.36.82.16:4301 , piece_id=VWKOS6JM7SSSSGRUYHJDVYQJLT645Z4NJYRF3TU4VXBVHSDID4OA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG, address:104.36.82.16:4301): protocol: write tcp 188.166.225.170:47256->104.36.82.16:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346187537 (84h44m6.068750577s) [noise=true , address=104.36.82.15:4285 , piece_id=NR246ARALGUBMGOUG7JAI4HP67PJ76RFWXDH2HFLAU4U2TBTUWMQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y, address:104.36.82.15:4285): protocol: rpc: dial tcp 104.36.82.15:4285: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346216452 (84h44m6.068779492s) [noise=true , address=104.36.82.53:4154 , piece_id=OQROWXS6GFWUNVE43TS6F3QMFXLIKSHEARAAYAV2WZJMPMMJJTQQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: rpc: dial tcp 104.36.82.53:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346243514 (84h44m6.068806574s) [noise=true , address=104.36.82.14:4256 , piece_id=AHYSCPLF24PX2GNEWXHE22WD5D7NY2OLMFWJGSO4VQJOKKWUUWVQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346274313 (84h44m6.068837353s) [noise=true , address=104.36.82.13:4220 , piece_id=QZYTN33PLPI5CBLOU6PBTXOYRTCOULZNQ5OCDQCJUO5WI5QO6EXQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1AjQmw3H58hTwr2389cmMcwy1gBavBrZB8AWB1gqfoFRUkCpH6, address:104.36.82.13:4220): protocol: write tcp 188.166.225.170:33164->104.36.82.13:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346321614 (84h44m6.068884654s) [noise=true , address=104.36.82.40:4973 , piece_id=UWOSZID6YLTHO54RBS4ZK3SFK2PA4OVTJB2VYY655KMFTTTAZ4VQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx, address:104.36.82.40:4973): protocol: rpc: dial tcp 104.36.82.40:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346328537 (84h44m6.068891577s) [noise=true , address=104.36.82.47:4531 , piece_id=4W3REBM7FHQXX5X4ITFFMZJ3X4LTMLY3CLZ7GAVZKN6YU47IQ2RA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346336002 (84h44m6.068899042s) [noise=true , address=104.36.82.11:4133 , piece_id=RVWCVPHJZ6JBTXH2N622GBCF3QH2KK6OFR2CE3GXWUE34ZVMR3PQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:12G5pngZWYQ2abKHEpM7Pa6YeCQ1LotX4qswbqHUwAy9H7a9Ym2, address:104.36.82.11:4133): protocol: rpc: dial tcp 104.36.82.11:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346343215 (84h44m6.068906255s) [noise=true , address=104.36.82.50:4032 , piece_id=DSXLYFSLOZY2PB73IYN2TDJWIXUGXMEPHDUWOGHJ3QR3FWV75UBA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1zj1d7Yzk4Z5Wda3YF6UgR5oSEhABcf6HCWwmHWXaX1CbKYJdQ, address:104.36.82.50:4032): protocol: rpc: dial tcp 104.36.82.50:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346358955 (84h44m6.068921985s) [noise=true , address=104.36.82.56:4246 , piece_id=NHHZS7SADGQTZSWUEM6PUYT7LMXKGSLIAEO4YZDZXXZTENWCVFFA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1XtKegw9zcPDGZu9z7STKgAzwx5KFrkB9qCgdNUe6XqZ6tzzj9, address:104.36.82.56:4246): protocol: rpc: dial tcp 104.36.82.56:4246: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346391077 (84h44m6.068954107s) [seg_pos={0 0}, upload=81307]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.238757793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 6.018631234s FAIL (2025-07-28 00:19:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.018651783s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X.......X.X] duration: 30.227579459s FAIL (2025-07-28 00:19:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.227604787s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................X............X..........] duration: 1.896335217s FAIL (2025-07-28 00:19:00 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.896362468s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..............................................X...XXXXXXXXXX] duration: 1m0.232186909s FAIL (2025-07-28 00:18:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== upload=81293]: Uploading piece... Jul 28 00:17:56.978157511 (84h42m6.700720551s) [noise=true , address=104.36.82.14:4224 , piece_id=2QUBF66SS3CDQZKT6NM6A6JKT7K3VQKA2JFF4ZV5SHFNMRCSPEOQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978329099 (84h42m6.700892129s) [noise=true , address=104.36.82.52:4115 , piece_id=H256VQY4J2E5XMLI2RJTWIATGKZFENZPMZ2YFIUJQQIK5YH5YFNQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978403171 (84h42m6.700966211s) [noise=true , address=104.36.82.16:4301 , piece_id=5IM4S3VGTBLDF6Y64OPFBZSZ3DVMGYQTOHQ4BYAL3OOFBKHIXMAA, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978629644 (84h42m6.701192684s) [noise=true , address=104.36.82.55:4222 , piece_id=A256DNPSGZBEOB3DNJW6PUAHAHGPWYH2HODCML5SSERCC4TTFUJQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978638331 (84h42m6.701201371s) [noise=true , address=104.36.82.12:4162 , piece_id=NXWDQ6LICRYIQ35NKFOM6JJC6PWAOUI532M72Y3GOWMQJLNIM74A, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978847832 (84h42m6.701410862s) [noise=true , address=104.36.82.41:4789 , piece_id=MZHEOD2O2BJHXXQN6FI37BCVOASM63LEVNGN5VHTTMQBVNSNRECQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978891915 (84h42m6.701454956s) [noise=true , address=104.36.82.44:4900 , piece_id=I7MGHZEOVXZQZ7RKSRXLGDKLU24Z43X7OZXX2VWR2AIOOFXF7USQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.979075247 (84h42m6.701638287s) [noise=true , address=79.127.220.101:4045 , piece_id=SXZUOR3GCKVPFYSKGXZC5DWB3PIMQPNJHEJHVIHSYZ4MSMPTUT7Q, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.979124190 (84h42m6.70168723s) [noise=true , address=104.36.82.10:4097 , piece_id=4GMFJ5URDVSABJREK3QXGGWWFVXL6WB67V726OPSF66WKGZFABSQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.979372385 (84h42m6.701935425s) [noise=true , address=79.127.201.215:4343 , piece_id=3PDQ3HFYAUEPDPMQ3N4QZ74TYSSCZZUUTV4D6JPO5YYKLW6JZREQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:58.073822806 (84h42m7.796385846s) [upload=81293]: Next returned nil segment Jul 28 00:17:58.073840519 (84h42m7.796403559s) [upload=81293]: Waiting for error group managing segments... Jul 28 00:17:58.358173788 (84h42m8.080736828s) [noise=true , address=79.127.205.235:4688 , piece_id=JGTIODKIDVXL6ZKRNKTBNS57YUUTRHM7KPM2M6G7WN2VFTGDXREQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: <nil> Jul 28 00:17:58.891902972 (84h42m8.614466012s) [noise=true , address=79.127.220.101:4045 , piece_id=SXZUOR3GCKVPFYSKGXZC5DWB3PIMQPNJHEJHVIHSYZ4MSMPTUT7Q, seg_pos={0 0}, upload=81293]: Done uploading piece. err: <nil> Jul 28 00:18:02.011626270 (84h42m11.73418931s) [noise=true , address=79.127.201.215:4343 , piece_id=3PDQ3HFYAUEPDPMQ3N4QZ74TYSSCZZUUTV4D6JPO5YYKLW6JZREQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: <nil> Jul 28 00:18:55.890590539 (84h43m5.613153569s) [noise=true , address=104.36.82.16:4301 , piece_id=5IM4S3VGTBLDF6Y64OPFBZSZ3DVMGYQTOHQ4BYAL3OOFBKHIXMAA, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG, address:104.36.82.16:4301): protocol: rpc: dial tcp 104.36.82.16:4301: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890635736 (84h43m5.613198766s) [noise=true , address=104.36.82.6:4898 , piece_id=PYH4GESFIJLJGM3SUF3CZKXFKD52K7TYWKIIVDHYJQ5XT5A255WA, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1D4E49RAbRK64c95rK6tGHPHHo5xLsmZ53ZK9i2iRU8mz7SbZ8, address:104.36.82.6:4898): protocol: rpc: dial tcp 104.36.82.6:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890667326 (84h43m5.613230366s) [noise=true , address=104.36.82.13:4213 , piece_id=4E7EVRG7XPF5D35VLKHFLVA3MO6F63OC5N6NI34QYS6XI2DJZQ4A, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:12XDcAySWNCL7XoStP5GKzAksN1kTs9L8u4KStgyaiQn3AvLcRM, address:104.36.82.13:4213): protocol: rpc: dial tcp 104.36.82.13:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890698335 (84h43m5.613261375s) [noise=true , address=104.36.82.44:4900 , piece_id=I7MGHZEOVXZQZ7RKSRXLGDKLU24Z43X7OZXX2VWR2AIOOFXF7USQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1R8K65zsCspu7W2oAnvtpz7zEsQFkDG1e5BAp6faQYMQrkh4oQ, address:104.36.82.44:4900): protocol: rpc: dial tcp 104.36.82.44:4900: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890728403 (84h43m5.613291443s) [noise=true , address=104.36.82.21:4507 , piece_id=FGZJDIKOHLMO2RWCMC5NJLMWRX5BP2RGYIEHO37LPSDVJQQMWXAA, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1XW3SaTU4fPuKGNeLtt4H9wQXw5ADEcvzcpo7J53wUULsMT5Bv, address:104.36.82.21:4507): protocol: rpc: dial tcp 104.36.82.21:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890758611 (84h43m5.613321641s) [noise=true , address=104.36.82.55:4222 , piece_id=A256DNPSGZBEOB3DNJW6PUAHAHGPWYH2HODCML5SSERCC4TTFUJQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:104.36.82.55:4222): protocol: write tcp 188.166.225.170:59390->104.36.82.55:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890783869 (84h43m5.613346909s) [noise=true , address=104.36.82.52:4115 , piece_id=H256VQY4J2E5XMLI2RJTWIATGKZFENZPMZ2YFIUJQQIK5YH5YFNQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890802234 (84h43m5.613365274s) [noise=true , address=104.36.82.54:4191 , piece_id=GMZKUFLZZ42LW77H3E4GLBX5IHAUBM75OGQ73NCDBI2PD3LAIAQQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: rpc: dial tcp 104.36.82.54:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890832552 (84h43m5.613395592s) [noise=true , address=104.36.82.53:4146 , piece_id=OVYPVDGILZG52JCJ35HADZ7KR3TOGDTTVR3NGAH4BRNVNRNKPSIQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 188.166.225.170:47596->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890848122 (84h43m5.613411162s) [noise=true , address=104.36.82.14:4224 , piece_id=2QUBF66SS3CDQZKT6NM6A6JKT7K3VQKA2JFF4ZV5SHFNMRCSPEOQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:104.36.82.14:4224): protocol: rpc: dial tcp 104.36.82.14:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890860345 (84h43m5.613423386s) [noise=true , address=104.36.82.41:4789 , piece_id=MZHEOD2O2BJHXXQN6FI37BCVOASM63LEVNGN5VHTTMQBVNSNRECQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1AXrW3TejqoDouALeZ1ddt8NqHZ5NZgZk5piqZHQNWgRjPHNqN, address:104.36.82.41:4789): protocol: rpc: dial tcp 104.36.82.41:4789: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890880544 (84h43m5.613443584s) [noise=true , address=104.36.82.12:4162 , piece_id=NXWDQ6LICRYIQ35NKFOM6JJC6PWAOUI532M72Y3GOWMQJLNIM74A, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1yxXQkfWVGwBuptpPjFgjhfTraAmE3WqtS3WEDbhQsJftjeaxV, address:104.36.82.12:4162): protocol: rpc: dial tcp 104.36.82.12:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890892126 (84h43m5.613455166s) [noise=true , address=104.36.82.5:4855 , piece_id=IW7RZRWZ2GN4FQKWYHAXDNKR2RSP5BMNHWQXCU74MCRG35FYQE7Q, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: rpc: dial tcp 104.36.82.5:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890922534 (84h43m5.613485574s) [noise=true , address=104.36.82.10:4097 , piece_id=4GMFJ5URDVSABJREK3QXGGWWFVXL6WB67V726OPSF66WKGZFABSQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3, address:104.36.82.10:4097): protocol: rpc: dial tcp 104.36.82.10:4097: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890931932 (84h43m5.613494962s) [noise=true , address=104.36.82.54:4169 , piece_id=HVCSJG4RPRBXSTKFZWYUDTSV7UAQQJUWLK65TNG44YZPDHINK3FQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890948454 (84h43m5.613511494s) [seg_pos={0 0}, upload=81293]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:18:56.120951736 (84h43m5.843514776s) [upload=81293]: Done uploading segments... Jul 28 00:18:56.120976824 (84h43m5.843539864s) [upload=81293]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.232201046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 5.871233103s FAIL (2025-07-28 00:18:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.871264923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.......X.] duration: 5.85053817s FAIL (2025-07-28 00:18:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.850560474s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 5.793558166s FAIL (2025-07-28 00:18:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.793579047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X............X.........] duration: 2.035964225s FAIL (2025-07-28 00:18:10 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.035991876s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.......X] duration: 30.230885213s FAIL (2025-07-28 00:18:02 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.230915011s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [...............................................X...XXXXXXXXX] duration: 1m0.234901758s FAIL (2025-07-28 00:17:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ce_id=V5DESVGF3MZF3Z7PSKJTLGANBLEYMQQMNQQRUZFUFBEMVWD3JFKQ, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.509982525 (84h41m6.232545565s) [noise=true , address=104.36.82.52:4087 , piece_id=YYOOIGB3SMO4QVW6PAZGO5HQANOECOZHBF77RJHRZCDZ3VF42RBQ, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.510173570 (84h41m6.2327366s) [noise=true , address=104.36.82.12:4164 , piece_id=P4YLWJZROD3CD2Q7M4U6ALX5Q2FCEJRKZEUWL2OUZUZJXBZSGS6A, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.510370828 (84h41m6.232933868s) [noise=true , address=104.36.82.10:4102 , piece_id=RYUNXCVFIBNYQKWVD6S7HTIREIP3AODZ3TUFGR7LQORA5IGR42FQ, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.511970599 (84h41m6.234533649s) [noise=true , address=104.36.82.50:4031 , piece_id=SXU73R2KHC2LAC2A3WELML3JBNHXVSQ4ZMO5U6X4UFE64AU22SHA, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.512187784 (84h41m6.234750824s) [noise=true , address=104.36.82.11:4141 , piece_id=KIAWZXCHYMGOKI4VMNNXJY7ONY6U5MLKSKARKDMD5FSJUSBVO6IQ, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.512474232 (84h41m6.235037272s) [noise=true , address=104.36.82.19:4420 , piece_id=TYQBC23IBRH6VXVZ2WQW2SDRUB4KKD6T3KCUQJYJ5XKQN63IDV7A, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:57.647297514 (84h41m7.369860544s) [upload=81277]: Next returned nil segment Jul 28 00:16:57.647316822 (84h41m7.369879862s) [upload=81277]: Waiting for error group managing segments... Jul 28 00:16:58.191648108 (84h41m7.914211148s) [noise=true , address=79.127.220.98:4404 , piece_id=IKZYNE7NQIF5CLRFX77WTOWTNTYB6JNFKXNNFA33LEVFD2T3BR4A, seg_pos={0 0}, upload=81277]: Done uploading piece. err: <nil> Jul 28 00:16:58.265274854 (84h41m7.987837894s) [noise=true , address=64.49.13.138:4483 , piece_id=UVACB7ENNG44VDUYUBV7RFPCZRNWM25LFHYM5UAGZYW7Y62MI7GQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: <nil> Jul 28 00:17:55.420737197 (84h42m5.143300237s) [noise=true , address=104.36.82.53:4145 , piece_id=V5DESVGF3MZF3Z7PSKJTLGANBLEYMQQMNQQRUZFUFBEMVWD3JFKQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: write tcp 188.166.225.170:57066->104.36.82.53:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420783546 (84h42m5.143346586s) [noise=true , address=104.36.82.50:4031 , piece_id=SXU73R2KHC2LAC2A3WELML3JBNHXVSQ4ZMO5U6X4UFE64AU22SHA, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 188.166.225.170:39058->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420815597 (84h42m5.143378637s) [noise=true , address=104.36.82.6:4893 , piece_id=D2SECKZBPZLUL4VRRIPNAWJZXPPT4HB46RXUGJYHYK5PFB7E5JSQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1259j7aHsLDMPtRwgXpd6u8axPEuaXzhRzqYYSgbpDnDqbhfwzi, address:104.36.82.6:4893): protocol: write tcp 188.166.225.170:50916->104.36.82.6:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420848761 (84h42m5.143411801s) [noise=true , address=104.36.82.40:4968 , piece_id=QMTLEGLL6LWCBZP7NWKTHBZJM77O6UIA4STGEVD6OG2YNMZWM3OQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1J98jLqn1SVMYjTYxKnNqYbRxSQ5vaUnq2G7XpVngHTaWRJ8oX, address:104.36.82.40:4968): protocol: write tcp 188.166.225.170:53122->104.36.82.40:4968: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420882876 (84h42m5.143445916s) [noise=true , address=104.36.82.11:4141 , piece_id=KIAWZXCHYMGOKI4VMNNXJY7ONY6U5MLKSKARKDMD5FSJUSBVO6IQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:4141): protocol: write tcp 188.166.225.170:56046->104.36.82.11:4141: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420916250 (84h42m5.14347929s) [noise=true , address=104.36.82.10:4102 , piece_id=RYUNXCVFIBNYQKWVD6S7HTIREIP3AODZ3TUFGR7LQORA5IGR42FQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1JEXRhcy1TphHqJXcd6qDv5gCdr6W2nJUcGoarZbsGt3TrtePy, address:104.36.82.10:4102): protocol: write tcp 188.166.225.170:34238->104.36.82.10:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420948652 (84h42m5.143511692s) [noise=true , address=104.36.82.51:4071 , piece_id=NFCGPCAKZVP3UXSSKQNJLJLAI5DVLPQRGLQVN4FZ7TDJZ4EZPXQQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1VqztUNDN3hhJAMvCJtPreW7BdtAWSofQxAtmDKfDiX5sNEfWY, address:104.36.82.51:4071): protocol: rpc: dial tcp 104.36.82.51:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420987597 (84h42m5.143550627s) [noise=true , address=104.36.82.14:4237 , piece_id=BUL7F2XHF3VJUQAAAOBTZU5KUIMFHYVOKJHI3IEPXKAJWD5XORRQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:104.36.82.14:4237): protocol: write tcp 188.166.225.170:54398->104.36.82.14:4237: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420997375 (84h42m5.143560415s) [noise=true , address=104.36.82.5:4847 , piece_id=XZQJ6QRZZWZ7K6ZEY4YUJUZTJNQGSKKX3GXBEE3YMZDJ7J4ARZZQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:4847): protocol: write tcp 188.166.225.170:35634->104.36.82.5:4847: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421036820 (84h42m5.14359986s) [noise=true , address=104.36.82.3:4799 , piece_id=MRKYWXL65FOM65XWLPDUFAZE7RSMNGE2I3AFFF6KH7GDDYB75FTA, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421052110 (84h42m5.14361515s) [noise=true , address=104.36.82.55:4213 , piece_id=WHJCETV7HXWYPI5DJRSG7WKVMTINQ5M4G5BRFD4SQMCDNTZU6DMA, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:104.36.82.55:4213): protocol: write tcp 188.166.225.170:36396->104.36.82.55:4213: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421080444 (84h42m5.143643484s) [noise=true , address=104.36.82.38:4720 , piece_id=QBD4TPG3ERF66Y6R4VTT2BICWRIOFGQU3BBGQIQV3COU7YXD2J5Q, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12Y16Z5H7jpsCyRzsAajF5AR3mURqTY8a4jgEr2xP4Yxy9W9kdC, address:104.36.82.38:4720): protocol: write tcp 188.166.225.170:58888->104.36.82.38:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421106795 (84h42m5.143669835s) [noise=true , address=104.36.82.52:4087 , piece_id=YYOOIGB3SMO4QVW6PAZGO5HQANOECOZHBF77RJHRZCDZ3VF42RBQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: write tcp 188.166.225.170:52952->104.36.82.52:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421149516 (84h42m5.143712556s) [noise=true , address=104.36.82.56:4258 , piece_id=A5STJ7KZVBDOUMVSEYXRA7FFKTWEQDWZMZDWGEZZMV6G5BD6D7MA, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12g7DdMjQHyYTD9hRvixfNedxyPr6q6uE2rLdEbhQ1MGANRMDm3, address:104.36.82.56:4258): protocol: write tcp 188.166.225.170:35374->104.36.82.56:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421155238 (84h42m5.143718288s) [noise=true , address=104.36.82.12:4164 , piece_id=P4YLWJZROD3CD2Q7M4U6ALX5Q2FCEJRKZEUWL2OUZUZJXBZSGS6A, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12iQgyDiz9vu1TbCRuATGmnrfbgBaia72b77ZjQJ6gpnEFbQzTJ, address:104.36.82.12:4164): protocol: write tcp 188.166.225.170:33080->104.36.82.12:4164: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421160457 (84h42m5.143723497s) [noise=true , address=104.36.82.19:4420 , piece_id=TYQBC23IBRH6VXVZ2WQW2SDRUB4KKD6T3KCUQJYJ5XKQN63IDV7A, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:104.36.82.19:4420): protocol: write tcp 188.166.225.170:42814->104.36.82.19:4420: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421176678 (84h42m5.143739718s) [seg_pos={0 0}, upload=81277]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:17:55.652349517 (84h42m5.374912557s) [upload=81277]: Done uploading segments... Jul 28 00:17:55.652384715 (84h42m5.374947755s) [upload=81277]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.234917607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 5.841081653s FAIL (2025-07-28 00:17:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.841103946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................X............X........] duration: 1.919648463s FAIL (2025-07-28 00:17:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.919680293s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [................................................X...XXXXXXXX] duration: 1m0.237180814s FAIL (2025-07-28 00:16:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== ul 28 00:15:56.038704237 (84h40m5.761267277s) [noise=true , address=104.36.82.53:4154 , piece_id=P7TTDUC6PM2KWCQCRCCILAKQ4HOX5F32BBD6SGY5ZP6GT4UJ3U3Q, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.038923066 (84h40m5.761486096s) [noise=true , address=104.36.82.56:4256 , piece_id=DV6NLET3AVZGL462JHC6RMRDM63GVDNSK7ZEJQD3N4RBV2QEDD4A, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.039139620 (84h40m5.76170266s) [noise=true , address=104.36.82.55:4205 , piece_id=66YVGGZEASCZS6IHWXCDC6DH67LR3NDQ2IJCBL67W3NKH7LYKP4A, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.039152615 (84h40m5.761715655s) [noise=true , address=104.36.82.22:4518 , piece_id=CA7NKROF666BTCMCNVW2K6HXWSQRBVIJP73H6FBBCURLELMVAADQ, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.039366424 (84h40m5.761929464s) [noise=true , address=104.36.82.11:4120 , piece_id=LNPKRA6D2O75O44CBPJKKER6ZE2Z52VXYXPV66KAUMOOMQUH5M6Q, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.039478889 (84h40m5.762041929s) [noise=true , address=104.36.82.9:4021 , piece_id=WRF3OMNR77GKWH5HNDGLQVQ4NS3CHSO623KUK3GF3BAXQCGKBJEQ, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.042284787 (84h40m5.764847827s) [noise=true , address=104.36.82.47:4532 , piece_id=KBYTMZOSQVJOUUM47VNFO5TRW6UJUVQQPUMWXP5P424A6PWQJZHQ, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.042642261 (84h40m5.765205301s) [noise=true , address=104.36.82.10:4101 , piece_id=LT53BM6ZPX7HXCKXTJSO7L4TSMRLOZNCVS3ODWJBY75553AL373A, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.042922227 (84h40m5.765485267s) [noise=true , address=64.49.13.140:4509 , piece_id=IYFIKRLJS6LLQ6T4ECNFTNI5FH4RP3GAK3AZOOIEIC6L4W7DSRTQ, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.043238372 (84h40m5.765801412s) [noise=true , address=104.36.82.54:4163 , piece_id=HH4JNYZES6QTFEWTMG26X3KSTKGUCZLZXNGVF6B3MARAXEN333HQ, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:57.198595100 (84h40m6.92115814s) [upload=81264]: Next returned nil segment Jul 28 00:15:57.198610891 (84h40m6.921173931s) [upload=81264]: Waiting for error group managing segments... Jul 28 00:15:57.690628138 (84h40m7.413191178s) [noise=true , address=64.49.13.140:4509 , piece_id=IYFIKRLJS6LLQ6T4ECNFTNI5FH4RP3GAK3AZOOIEIC6L4W7DSRTQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: <nil> Jul 28 00:15:58.353679667 (84h40m8.076242708s) [noise=true , address=64.49.13.138:4482 , piece_id=IAOXAL7D4JGTLA3YME7UZADCHYFHFLZYPBO4WV7NN7F55P366MNA, seg_pos={0 0}, upload=81264]: Done uploading piece. err: <nil> Jul 28 00:15:59.537815957 (84h40m9.260378997s) [noise=true , address=79.127.201.215:4346 , piece_id=C5JV4YHK5T2EJPNFP544PSVFJ34PKGRYE35BCE3TSN7EUYPAE5AQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: <nil> Jul 28 00:16:54.937395756 (84h41m4.659958797s) [noise=true , address=104.36.82.15:4277 , piece_id=DUMG7WOWJPYRH3OJIMGEJYI3FVCLU2ZA2LWEVNLRDJFBBREBFUSQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:1i493KQZuQYQwXzct9HamWDCq92CSGvLPHjZ6fxxVVZX1f7ofY, address:104.36.82.15:4277): protocol: rpc: dial tcp 104.36.82.15:4277: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937446704 (84h41m4.660009734s) [noise=true , address=104.36.82.55:4197 , piece_id=DGDTNLKYOEWE5RI27BVDY5PQZC64ANIY2DQL7VZSG4KJDWOR53DA, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: rpc: dial tcp 104.36.82.55:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937477763 (84h41m4.660040813s) [noise=true , address=104.36.82.46:4490 , piece_id=P7YYMWSRWBN2C6PE52FLZY5GGWZXCLBVBAQS77JTUV6E63UMLXFQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: rpc: dial tcp 104.36.82.46:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937510205 (84h41m4.660073245s) [noise=true , address=104.36.82.53:4154 , piece_id=P7TTDUC6PM2KWCQCRCCILAKQ4HOX5F32BBD6SGY5ZP6GT4UJ3U3Q, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: rpc: dial tcp 104.36.82.53:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937576563 (84h41m4.660139603s) [noise=true , address=104.36.82.55:4205 , piece_id=66YVGGZEASCZS6IHWXCDC6DH67LR3NDQ2IJCBL67W3NKH7LYKP4A, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ, address:104.36.82.55:4205): protocol: rpc: dial tcp 104.36.82.55:4205: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937642859 (84h41m4.660205899s) [noise=true , address=104.36.82.4:4813 , piece_id=OJUKKE66EHBQOEOQQ4CKWNOFAXMESJP7YVUNK73FZRK7OG6WOKIA, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:104.36.82.4:4813): protocol: write tcp 188.166.225.170:48730->104.36.82.4:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937679138 (84h41m4.660242178s) [noise=true , address=104.36.82.50:4015 , piece_id=DBKNZBUPKKWIITIGDWU4CHCLQ2IS3PAMUVBKWLD47GSBIIEXUV4Q, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:104.36.82.50:4015): protocol: rpc: dial tcp 104.36.82.50:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937716340 (84h41m4.66027937s) [noise=true , address=104.36.82.36:4634 , piece_id=I24D6SC22IPDGYQHC5MR7N4H4UXPNSUGFC2NMXOWCWID3U4GXJOQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:1kwydp3q2wTrCVadJZB7WmKGK8h2R5QL7HGpUsjTpRbcnWUSb5, address:104.36.82.36:4634): protocol: rpc: dial tcp 104.36.82.36:4634: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937746518 (84h41m4.660309558s) [noise=true , address=104.36.82.10:4101 , piece_id=LT53BM6ZPX7HXCKXTJSO7L4TSMRLOZNCVS3ODWJBY75553AL373A, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937838624 (84h41m4.660401664s) [noise=true , address=104.36.82.54:4163 , piece_id=HH4JNYZES6QTFEWTMG26X3KSTKGUCZLZXNGVF6B3MARAXEN333HQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: write tcp 188.166.225.170:56394->104.36.82.54:4163: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937879693 (84h41m4.660442733s) [noise=true , address=104.36.82.56:4256 , piece_id=DV6NLET3AVZGL462JHC6RMRDM63GVDNSK7ZEJQD3N4RBV2QEDD4A, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: rpc: dial tcp 104.36.82.56:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937901905 (84h41m4.660464935s) [noise=true , address=104.36.82.9:4021 , piece_id=WRF3OMNR77GKWH5HNDGLQVQ4NS3CHSO623KUK3GF3BAXQCGKBJEQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12ts2cPQYUxdscD7RBpVxiHk36KAtJnXnAP9FwY7gCpBXHeS1Cg, address:104.36.82.9:4021): protocol: rpc: dial tcp 104.36.82.9:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937918316 (84h41m4.660481356s) [noise=true , address=104.36.82.22:4518 , piece_id=CA7NKROF666BTCMCNVW2K6HXWSQRBVIJP73H6FBBCURLELMVAADQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:1ozLDHDa9ankBimUBjpM91VrraKMptSCQPoDS9yWu2tjVErasU, address:104.36.82.22:4518): protocol: rpc: dial tcp 104.36.82.22:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937946160 (84h41m4.6605092s) [noise=true , address=104.36.82.47:4532 , piece_id=KBYTMZOSQVJOUUM47VNFO5TRW6UJUVQQPUMWXP5P424A6PWQJZHQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937972520 (84h41m4.66053556s) [noise=true , address=104.36.82.11:4120 , piece_id=LNPKRA6D2O75O44CBPJKKER6ZE2Z52VXYXPV66KAUMOOMQUH5M6Q, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:123jcA328VCSUaWMKvSaeXip1U6Sc4SSvuRaoHF8fvsF6Tt6DHW, address:104.36.82.11:4120): protocol: write tcp 188.166.225.170:55330->104.36.82.11:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937998750 (84h41m4.66056179s) [seg_pos={0 0}, upload=81264]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:16:55.172427108 (84h41m4.894990148s) [upload=81264]: Done uploading segments... Jul 28 00:16:55.172457186 (84h41m4.895020226s) [upload=81264]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.237213348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 5.831422759s FAIL (2025-07-28 00:16:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.83144999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.008775427s FAIL (2025-07-28 00:16:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.00880365s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X............X.......] duration: 1.905069773s FAIL (2025-07-28 00:16:19 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.905096363s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 5.876948712s FAIL (2025-07-28 00:15:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.876972637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.................................................X...XXXXXXX] duration: 1m0.229436487s FAIL (2025-07-28 00:15:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== : Uploading piece... Jul 28 00:14:53.331896307 (84h39m3.054459347s) [noise=true , address=104.36.82.46:4488 , piece_id=4S57UPCVJQGDDJW6EEJMKTEJNDXUUDCX4UCDWTROVDUPF6TVOLUA, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.332130846 (84h39m3.054693886s) [noise=true , address=104.36.82.42:4812 , piece_id=KHVC7T4LZBS5EFCM3NUY3BGAGLYKSSXHIFADD3XZ57SOLXM4SUJA, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.332396654 (84h39m3.054959694s) [noise=true , address=104.36.82.9:4032 , piece_id=GNBIEEQ2FDSZYIHGASPQ5QHYMMK7PQTM7URHIHBWQDLECI6SQRSA, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.332618228 (84h39m3.055181268s) [noise=true , address=104.36.82.4:4828 , piece_id=EZIFSYBIGFQFTCNMFFGHG5XYIT2UO6SVMZBH4PUGMPZDHCE3VVVQ, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.332842558 (84h39m3.055405588s) [noise=true , address=104.36.82.49:4990 , piece_id=FCXU7726KRDGV2ZMIWOCMNFO2PRBY4GAFLXK6PLV6KM2XODFYYBQ, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.333070313 (84h39m3.055633353s) [noise=true , address=104.36.82.15:4279 , piece_id=R4IDM5KBBZKVLKDVFCTUH4FYI7JHJEEVWEXMXCZE3OEV5DCT5CJQ, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.333287098 (84h39m3.055850128s) [noise=true , address=104.36.82.13:4188 , piece_id=K2ZWCGD2ZJU7RL6Z3NV3LZ3Z7OA62F6ULRUWDHGVWEGLUOZU4IXA, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.333504775 (84h39m3.056067815s) [noise=true , address=104.36.82.55:4201 , piece_id=MRRUQF7KYU3ZDKWEYDSMTAYY5HFSFY6UFFW7TSAP2J6GDPITG2AQ, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:54.841644854 (84h39m4.564207894s) [upload=81249]: Next returned nil segment Jul 28 00:14:54.841663019 (84h39m4.564226059s) [upload=81249]: Waiting for error group managing segments... Jul 28 00:14:55.226766917 (84h39m4.949329957s) [noise=true , address=79.127.201.214:4291 , piece_id=MAF5NZGDQY64EMZXE7KTO76PQLE4EDAGY642BXCKC4FWFKFGCPEA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: <nil> Jul 28 00:14:56.329867134 (84h39m6.052430175s) [noise=true , address=64.49.13.140:4508 , piece_id=6INZQBFHYBJFICMBZB6ZNNH225TO6LRZ65JTAJRBLD6KE3ZJCOVA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: <nil> Jul 28 00:15:52.240213723 (84h40m1.962776763s) [noise=true , address=104.36.82.5:4842 , piece_id=BQACPUFGZCKZHCTZRRXE77KNMIFLOVCJIGHDJGV3WDYPWNYQB6SQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240287545 (84h40m1.962850585s) [noise=true , address=104.36.82.47:4528 , piece_id=DULYM42F56QCMRCAJPQHC4I3SNMQE3YEDA5UEEY7TCEJ2SXSBWVQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: rpc: dial tcp 104.36.82.47:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240507244 (84h40m1.963070284s) [noise=true , address=104.36.82.14:4240 , piece_id=RT47AXFOZSGNOOAGUHG6U4BBRNL3ZTTWMAS7OEZ33F7RQPVDTAHA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:13SaC5StbguPT3abAT9oTX3mZAyGhJcpAA6b7ueyiDFHa8CCEo, address:104.36.82.14:4240): protocol: rpc: dial tcp 104.36.82.14:4240: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240566037 (84h40m1.963129108s) [noise=true , address=104.36.82.12:4184 , piece_id=O2OHDSYBKHHF3ZCSVBYEHGVDO3N24BFXWUNPUDELV52ZACXOCFHA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:17JJjwwgSn7A6BS7n1a7fphX8oTEQpXmad25ifM2nDhWhPv4Ls, address:104.36.82.12:4184): protocol: rpc: dial tcp 104.36.82.12:4184: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240616984 (84h40m1.963180034s) [noise=true , address=104.36.82.4:4828 , piece_id=EZIFSYBIGFQFTCNMFFGHG5XYIT2UO6SVMZBH4PUGMPZDHCE3VVVQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: write tcp 188.166.225.170:46712->104.36.82.4:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240685956 (84h40m1.963248997s) [noise=true , address=104.36.82.42:4812 , piece_id=KHVC7T4LZBS5EFCM3NUY3BGAGLYKSSXHIFADD3XZ57SOLXM4SUJA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:1TmXeVTeejZvrWi2RPxMutM6kXuZiMCtPxQBwL4tq6mhoLwFQo, address:104.36.82.42:4812): protocol: write tcp 188.166.225.170:53524->104.36.82.42:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240730933 (84h40m1.963293973s) [noise=true , address=104.36.82.46:4502 , piece_id=IK6H4XUBDSC27HNUSCAYS7D3OXBMCLIARPPCFTMB6H5XDS225DCA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240886610 (84h40m1.96344965s) [noise=true , address=104.36.82.3:4799 , piece_id=VMYQUXIGXCPHFFPAYMBTIBKQ6SDN62NO6IU2C66RESUUHOZNNRTQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: rpc: dial tcp 104.36.82.3:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241047739 (84h40m1.963610779s) [noise=true , address=104.36.82.45:4936 , piece_id=PCTADNWF4UUA27YJBIRMNWCGHOIRKA7ZGRNAH4DPQHSMI72NOBGA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8, address:104.36.82.45:4936): protocol: rpc: dial tcp 104.36.82.45:4936: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241141237 (84h40m1.963704287s) [noise=true , address=104.36.82.15:4279 , piece_id=R4IDM5KBBZKVLKDVFCTUH4FYI7JHJEEVWEXMXCZE3OEV5DCT5CJQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: write tcp 188.166.225.170:56188->104.36.82.15:4279: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241230919 (84h40m1.963793959s) [noise=true , address=104.36.82.9:4032 , piece_id=GNBIEEQ2FDSZYIHGASPQ5QHYMMK7PQTM7URHIHBWQDLECI6SQRSA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ, address:104.36.82.9:4032): protocol: write tcp 188.166.225.170:52692->104.36.82.9:4032: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241269803 (84h40m1.963832853s) [noise=true , address=104.36.82.54:4182 , piece_id=7JBUEBASEJEFJ7HIIU2OED56QSN4REZEUZLAMXQNTEVHOXEMPMXQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12HbkH31sfDUnZLXAbmzQphcFYmzB8MQo98Q6Ufbo59et42zxMy, address:104.36.82.54:4182): protocol: rpc: dial tcp 104.36.82.54:4182: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241310532 (84h40m1.963873582s) [noise=true , address=104.36.82.13:4188 , piece_id=K2ZWCGD2ZJU7RL6Z3NV3LZ3Z7OA62F6ULRUWDHGVWEGLUOZU4IXA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12pLFHQ2Xj6n53w9eLQtMB4A29k3Xh6w716z5s2CGxwPnBoZcgZ, address:104.36.82.13:4188): protocol: rpc: dial tcp 104.36.82.13:4188: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241319639 (84h40m1.963882679s) [noise=true , address=104.36.82.46:4488 , piece_id=4S57UPCVJQGDDJW6EEJMKTEJNDXUUDCX4UCDWTROVDUPF6TVOLUA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: write tcp 188.166.225.170:35244->104.36.82.46:4488: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241353343 (84h40m1.963916393s) [noise=true , address=104.36.82.49:4990 , piece_id=FCXU7726KRDGV2ZMIWOCMNFO2PRBY4GAFLXK6PLV6KM2XODFYYBQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:1gDzCKcMaLTisHw68GYkMbcMJF7rbKnUt48etS8Q7KeiXedbU4, address:104.36.82.49:4990): protocol: write tcp 188.166.225.170:35316->104.36.82.49:4990: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241372310 (84h40m1.96393535s) [noise=true , address=104.36.82.55:4201 , piece_id=MRRUQF7KYU3ZDKWEYDSMTAYY5HFSFY6UFFW7TSAP2J6GDPITG2AQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: write tcp 188.166.225.170:56302->104.36.82.55:4201: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241392449 (84h40m1.963955499s) [seg_pos={0 0}, upload=81249]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:15:52.467954056 (84h40m2.190517106s) [upload=81249]: Done uploading segments... Jul 28 00:15:52.467984765 (84h40m2.190547805s) [upload=81249]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.229450765s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)