Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....] duration: 6.79362009s FAIL (2025-07-27 00:50:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.793663814s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [X.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X......] duration: 7.585960884s FAIL (2025-07-27 00:50:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.585996832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.883527856s FAIL (2025-07-27 00:50:07 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.883555178s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.XX...XXXX.....X..........X..X..X...XX...................] duration: 5.825476586s FAIL (2025-07-27 00:50:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.825503427s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..X...............XX......X...XXX..XXX..X..X.X..............] duration: 5.438780472s FAIL (2025-07-27 00:49:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.438808275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X...] duration: 5.747184156s FAIL (2025-07-27 00:49:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.747212039s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration: 9.448880439s FAIL (2025-07-27 00:49:19 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 9.448903184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X.....] duration: 6.178562075s FAIL (2025-07-27 00:49:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.178586371s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.XX...XXXX.....X..........X..X..X...XX..................] duration: 6.253792535s FAIL (2025-07-27 00:49:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.253813686s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 5.660656678s FAIL (2025-07-27 00:48:58 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 5.660676826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.123342153s FAIL (2025-07-27 00:48:55 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.123368403s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 54.988237073s FAIL (2025-07-27 00:48:52 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 54.988265047s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...X...............XX......X...XXX..XXX..X..X.X.............] duration: 5.97440813s FAIL (2025-07-27 00:48:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.974430013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [X.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X..] duration: 6.013295725s FAIL (2025-07-27 00:48:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.013322154s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.XX...XXXX.....X..........X..X..X...XX.................] duration: 6.180230386s FAIL (2025-07-27 00:48:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.180270542s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X....] duration: 5.737316725s FAIL (2025-07-27 00:48:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.737345119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.870581915s FAIL (2025-07-27 00:48:03 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.870604518s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 40 out of 60 is failed (threshold=3) [XX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X.] duration: 5.670489606s FAIL (2025-07-27 00:47:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.670510095s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X.XX...XXXX.....X..........X..X..X...XX................] duration: 5.938616572s FAIL (2025-07-27 00:47:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.938643554s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X...X...............XX......X...XXX..XXX..X..X.X............] duration: 5.484417419s FAIL (2025-07-27 00:47:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.484446394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.828984331s FAIL (2025-07-27 00:47:39 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.829022104s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 6.099561864s FAIL (2025-07-27 00:47:32 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 6.099585509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X] duration: 30.251678107s FAIL (2025-07-27 00:47:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.251710057s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X...] duration: 5.862064133s FAIL (2025-07-27 00:47:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.862106754s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X.XX...XXXX.....X..........X..X..X...XX...............] duration: 6.270812105s FAIL (2025-07-27 00:46:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.270851551s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XX...X...............XX......X...XXX..XXX..X..X.X...........] duration: 11.21744075s FAIL (2025-07-27 00:46:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.217466159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.11253865s FAIL (2025-07-27 00:46:30 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.112563328s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 5.844493014s FAIL (2025-07-27 00:46:18 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 5.844519064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..] duration: 5.877582234s FAIL (2025-07-27 00:46:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.877608424s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.XX...X...............XX......X...XXX..XXX..X..X.X..........] duration: 5.841733596s FAIL (2025-07-27 00:45:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.841756099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X......X.XX...XXXX.....X..........X..X..X...XX..............] duration: 6.070311211s FAIL (2025-07-27 00:45:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.07033262s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X.] duration: 5.990721693s FAIL (2025-07-27 00:45:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.990752801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.887408481s FAIL (2025-07-27 00:45:47 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.887429471s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 40 out of 60 is failed (threshold=3) [.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....] duration: 5.89256497s FAIL (2025-07-27 00:45:43 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.892594706s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 23.515044542s FAIL (2025-07-27 00:45:39 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 23.515071704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [X.XX...X...............XX......X...XXX..XXX..X..X.X.........] duration: 15.068867271s FAIL (2025-07-27 00:45:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.068887769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.065602369s FAIL (2025-07-27 00:45:29 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.065636192s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X......X.XX...XXXX.....X..........X..X..X...XX.............] duration: 6.097296876s FAIL (2025-07-27 00:45:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.09731974s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX....] duration: 5.784766461s FAIL (2025-07-27 00:45:13 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.784824352s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X] duration: 30.243940932s FAIL (2025-07-27 00:45:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.243963125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.823529193s FAIL (2025-07-27 00:44:52 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.823560473s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X......X.XX...XXXX.....X..........X..X..X...XX............] duration: 6.189327774s FAIL (2025-07-27 00:44:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.189349926s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX...] duration: 5.759593198s FAIL (2025-07-27 00:44:38 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.759614718s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 5.589055217s FAIL (2025-07-27 00:44:30 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 5.589084654s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X......X.XX...XXXX.....X..........X..X..X...XX...........] duration: 6.188575553s FAIL (2025-07-27 00:44:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.188597895s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.002624114s FAIL (2025-07-27 00:44:16 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 1m0.002651076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [X.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX..] duration: 5.990545205s FAIL (2025-07-27 00:44:16 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.990572557s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.X.XX...X...............XX......X...XXX..XXX..X..X.X........] duration: 5.677749798s FAIL (2025-07-27 00:44:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.677772321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.] duration: 6.991306945s FAIL (2025-07-27 00:44:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== loading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:53950->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:07.609111953 (61h7m17.331674993s) [seg_pos={0 0}, upload=58605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:43:07.609198008 (61h7m17.331761038s) [upload=58605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:43:07.609202506 (61h7m17.331765546s) [upload=58605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:43:07.896913209 (61h7m17.619476249s) [upload=58605]: Done uploading segments... Jul 27 00:43:07.896931484 (61h7m17.619494524s) [upload=58605]: Upload finished. err: <nil> Jul 27 00:43:28.060523287 (61h7m37.783086327s) [upload=58612]: Starting upload Jul 27 00:43:28.060583983 (61h7m37.783147013s) [upload=58612]: Done starting upload Jul 27 00:43:28.060670480 (61h7m37.78323352s) [upload=58612]: Uploading segments... Jul 27 00:43:28.060757105 (61h7m37.783320135s) [upload=58612]: Got next segment. Inline: false Jul 27 00:43:28.060782454 (61h7m37.783345484s) [upload=58612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:28.060787423 (61h7m37.783350463s) [upload=58612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:28.290711414 (61h7m38.013274534s) [seg_pos={0 0}, upload=58612]: Begin upload segment... Jul 27 00:43:28.290781437 (61h7m38.013344487s) [seg_pos={0 0}, upload=58612]: Done begin upload segment. Jul 27 00:43:28.290806114 (61h7m38.013369154s) [noise=true , address=45.91.116.4:4940 , piece_id=MA45FUZBITCY6UR2CJFIEQGBZW2ZITG4AIED54LXO2SMMHJMPHJA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.290862703 (61h7m38.013425763s) [noise=true , address=45.91.116.6:4975 , piece_id=M2FD6AJSLFQLE4LCUG7EL5MP7LVBRA4CB2M4N3TADFQSFJV7QWPQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291198386 (61h7m38.013761426s) [noise=true , address=104.36.82.2:4732 , piece_id=HFCA7UMY5YMFNDI4F26QTR6DXTD3XZBOQV4BZBBRCK7SQMBDS2YQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291275082 (61h7m38.013838132s) [noise=true , address=77.90.31.13:4628 , piece_id=BLOWAKSVHXH7KUJVXZY6ZRX5MZWH3SUEQR2OO4Z6JKNCBGNM7VUQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291495654 (61h7m38.014058704s) [noise=true , address=45.91.116.14:4191 , piece_id=CHDM6NXETFACDXIUPRKQPYWJD4WDQDRDQOMHGF3KASZ5KDF2UHJQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291856834 (61h7m38.014419864s) [noise=true , address=45.91.116.13:4155 , piece_id=L2EBEYRWI4UZTNIHKRNCDY4ZI6Y5ZRKZCG4VUKRBSFCE6PQED7UA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291880410 (61h7m38.01444347s) [noise=true , address=104.36.82.12:4149 , piece_id=JVI4QFQMSNIJNX2LEWWDHZ3RPSMGY63326OXOFIQCZOCXM2DTN7A, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292322225 (61h7m38.014885265s) [noise=true , address=104.36.82.9:4005 , piece_id=XNVTMCWLML6MZLHYO7AF2N35B6FJZBUOH5T6WPG42Y4I5XCUHSZA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292451232 (61h7m38.015014282s) [noise=true , address=104.36.82.7:4911 , piece_id=DTUXFUXCMBDBNPMPS6FVF4DWLKZWKAX2MNAB36EZTHHVJ3F6ORSQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292763219 (61h7m38.015326259s) [noise=true , address=45.91.116.13:4153 , piece_id=MWCNTPBJRJFE4NVHAXNIH7BV3K3UD3O7Q6NIXFQ3YQYKUWEGVB6Q, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292937924 (61h7m38.015500964s) [noise=true , address=45.91.116.8:4045 , piece_id=ZYU4527GKVC6ZIDR4AH2TXGAN6VC63QFKHOJZJNQQ32FBAGTXOAQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293180598 (61h7m38.015743638s) [noise=true , address=104.36.82.5:4840 , piece_id=YTP7X34D2JFIN23IMU2FG2PRDJZMVG4RNDW2LUATWPQW3YF2REVQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293528122 (61h7m38.016091172s) [noise=true , address=104.36.82.13:4186 , piece_id=5BZ75HYCFMRX5XSUHPCG3KBWHQ7TXF5ZLBZSTKF2OLHI7O7CE52Q, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293599289 (61h7m38.016162329s) [noise=true , address=77.90.31.16:4733 , piece_id=UN56NU4BYL3ZJCYRTL3V4FXNE2E6XIOXQL6I3X2LTUBVF3GJIBYQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.294088004 (61h7m38.016651044s) [noise=true , address=104.36.82.10:4077 , piece_id=IC6VVDBP4VMLHV57GCLNSTXXAY44QT2WMV7Y733IC56CPAZ4ZDAA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.298318578 (61h7m38.020881608s) [noise=true , address=77.90.31.11:4555 , piece_id=VACZCFH53LME27B77R5P6JJLPWTXGFJ5E3UDYEEWFKIKHRKJDYVA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:30.041308206 (61h7m39.763871246s) [upload=58612]: Next returned nil segment Jul 27 00:43:30.041324637 (61h7m39.763887677s) [upload=58612]: Waiting for error group managing segments... Jul 27 00:43:30.349307817 (61h7m40.071870857s) [noise=true , address=104.36.82.9:4005 , piece_id=XNVTMCWLML6MZLHYO7AF2N35B6FJZBUOH5T6WPG42Y4I5XCUHSZA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.375421520 (61h7m40.09798456s) [noise=true , address=45.91.116.4:4940 , piece_id=MA45FUZBITCY6UR2CJFIEQGBZW2ZITG4AIED54LXO2SMMHJMPHJA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.381080437 (61h7m40.103643477s) [noise=true , address=45.91.116.13:4153 , piece_id=MWCNTPBJRJFE4NVHAXNIH7BV3K3UD3O7Q6NIXFQ3YQYKUWEGVB6Q, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.411307720 (61h7m40.13387076s) [noise=true , address=45.91.116.8:4045 , piece_id=ZYU4527GKVC6ZIDR4AH2TXGAN6VC63QFKHOJZJNQQ32FBAGTXOAQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.454531159 (61h7m40.177094199s) [noise=true , address=45.91.116.14:4191 , piece_id=CHDM6NXETFACDXIUPRKQPYWJD4WDQDRDQOMHGF3KASZ5KDF2UHJQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.499012215 (61h7m40.221575255s) [noise=true , address=45.91.116.13:4155 , piece_id=L2EBEYRWI4UZTNIHKRNCDY4ZI6Y5ZRKZCG4VUKRBSFCE6PQED7UA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.522553385 (61h7m40.245116425s) [noise=true , address=104.36.82.2:4732 , piece_id=HFCA7UMY5YMFNDI4F26QTR6DXTD3XZBOQV4BZBBRCK7SQMBDS2YQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.553051998 (61h7m40.275615038s) [noise=true , address=45.91.116.6:4975 , piece_id=M2FD6AJSLFQLE4LCUG7EL5MP7LVBRA4CB2M4N3TADFQSFJV7QWPQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:31.247387600 (61h7m40.96995063s) [noise=true , address=104.36.82.5:4840 , piece_id=YTP7X34D2JFIN23IMU2FG2PRDJZMVG4RNDW2LUATWPQW3YF2REVQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:32.097194573 (61h7m41.819757653s) [noise=true , address=77.90.31.16:4733 , piece_id=UN56NU4BYL3ZJCYRTL3V4FXNE2E6XIOXQL6I3X2LTUBVF3GJIBYQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:32.214252718 (61h7m41.936815758s) [noise=true , address=77.90.31.11:4555 , piece_id=VACZCFH53LME27B77R5P6JJLPWTXGFJ5E3UDYEEWFKIKHRKJDYVA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:57.025053300 (61h8m6.74761634s) [noise=true , address=77.90.31.13:4628 , piece_id=BLOWAKSVHXH7KUJVXZY6ZRX5MZWH3SUEQR2OO4Z6JKNCBGNM7VUQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:57.418787467 (61h8m7.141350507s) [noise=true , address=104.36.82.10:4077 , piece_id=IC6VVDBP4VMLHV57GCLNSTXXAY44QT2WMV7Y733IC56CPAZ4ZDAA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:35220->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418925340 (61h8m7.14148838s) [noise=true , address=104.36.82.7:4911 , piece_id=DTUXFUXCMBDBNPMPS6FVF4DWLKZWKAX2MNAB36EZTHHVJ3F6ORSQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:55892->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418975867 (61h8m7.141538907s) [noise=true , address=104.36.82.13:4186 , piece_id=5BZ75HYCFMRX5XSUHPCG3KBWHQ7TXF5ZLBZSTKF2OLHI7O7CE52Q, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:52952->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418980546 (61h8m7.141543586s) [noise=true , address=104.36.82.12:4149 , piece_id=JVI4QFQMSNIJNX2LEWWDHZ3RPSMGY63326OXOFIQCZOCXM2DTN7A, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:43978->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.419029480 (61h8m7.14159256s) [seg_pos={0 0}, upload=58612]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:43:57.648786891 (61h8m7.371349931s) [upload=58612]: Done uploading segments... Jul 27 00:43:57.648814473 (61h8m7.371377514s) [upload=58612]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.991327305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.977039922s FAIL (2025-07-27 00:44:02 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.977067905s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)