Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X....] duration: 5.737316725s FAIL (2025-07-27 00:48:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.737345119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.870581915s FAIL (2025-07-27 00:48:03 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.870604518s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 40 out of 60 is failed (threshold=3) [XX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X.] duration: 5.670489606s FAIL (2025-07-27 00:47:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.670510095s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X.XX...XXXX.....X..........X..X..X...XX................] duration: 5.938616572s FAIL (2025-07-27 00:47:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.938643554s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X...X...............XX......X...XXX..XXX..X..X.X............] duration: 5.484417419s FAIL (2025-07-27 00:47:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.484446394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.828984331s FAIL (2025-07-27 00:47:39 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.829022104s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 6.099561864s FAIL (2025-07-27 00:47:32 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 6.099585509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X] duration: 30.251678107s FAIL (2025-07-27 00:47:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.251710057s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X...] duration: 5.862064133s FAIL (2025-07-27 00:47:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.862106754s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X.XX...XXXX.....X..........X..X..X...XX...............] duration: 6.270812105s FAIL (2025-07-27 00:46:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.270851551s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XX...X...............XX......X...XXX..XXX..X..X.X...........] duration: 11.21744075s FAIL (2025-07-27 00:46:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.217466159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.11253865s FAIL (2025-07-27 00:46:30 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.112563328s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 5.844493014s FAIL (2025-07-27 00:46:18 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 5.844519064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..] duration: 5.877582234s FAIL (2025-07-27 00:46:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.877608424s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.XX...X...............XX......X...XXX..XXX..X..X.X..........] duration: 5.841733596s FAIL (2025-07-27 00:45:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.841756099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X......X.XX...XXXX.....X..........X..X..X...XX..............] duration: 6.070311211s FAIL (2025-07-27 00:45:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.07033262s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X.] duration: 5.990721693s FAIL (2025-07-27 00:45:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.990752801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.887408481s FAIL (2025-07-27 00:45:47 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.887429471s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 40 out of 60 is failed (threshold=3) [.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....] duration: 5.89256497s FAIL (2025-07-27 00:45:43 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.892594706s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 23.515044542s FAIL (2025-07-27 00:45:39 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 23.515071704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [X.XX...X...............XX......X...XXX..XXX..X..X.X.........] duration: 15.068867271s FAIL (2025-07-27 00:45:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.068887769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.065602369s FAIL (2025-07-27 00:45:29 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.065636192s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X......X.XX...XXXX.....X..........X..X..X...XX.............] duration: 6.097296876s FAIL (2025-07-27 00:45:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.09731974s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX....] duration: 5.784766461s FAIL (2025-07-27 00:45:13 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.784824352s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X] duration: 30.243940932s FAIL (2025-07-27 00:45:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.243963125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.823529193s FAIL (2025-07-27 00:44:52 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.823560473s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X......X.XX...XXXX.....X..........X..X..X...XX............] duration: 6.189327774s FAIL (2025-07-27 00:44:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.189349926s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX...] duration: 5.759593198s FAIL (2025-07-27 00:44:38 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.759614718s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 5.589055217s FAIL (2025-07-27 00:44:30 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 5.589084654s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X......X.XX...XXXX.....X..........X..X..X...XX...........] duration: 6.188575553s FAIL (2025-07-27 00:44:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.188597895s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.002624114s FAIL (2025-07-27 00:44:16 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== : Got next segment. Inline: false Jul 27 00:43:16.823062814 (61h7m26.545625895s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:16.823071120 (61h7m26.54563419s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:17.263877438 (61h7m26.986440478s) [seg_pos={0 0}, upload=58607]: Begin upload segment... Jul 27 00:43:17.263932604 (61h7m26.986495654s) [seg_pos={0 0}, upload=58607]: Done begin upload segment. Jul 27 00:43:17.263961639 (61h7m26.986524689s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.263996126 (61h7m26.986559216s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264445626 (61h7m26.987008666s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264699070 (61h7m26.9872621s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.264836503 (61h7m26.987399553s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265207143 (61h7m26.987770193s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265211952 (61h7m26.987774992s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265589014 (61h7m26.988152054s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265668997 (61h7m26.988232037s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.265909556 (61h7m26.988472596s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266195424 (61h7m26.988758464s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266328337 (61h7m26.988891377s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266629213 (61h7m26.989192253s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266694839 (61h7m26.989257869s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.266934117 (61h7m26.989497167s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267133709 (61h7m26.989696749s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267276141 (61h7m26.989839191s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:17.267573941 (61h7m26.990136981s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Uploading piece... Jul 27 00:43:18.382676321 (61h7m28.105239361s) [upload=58607]: Next returned nil segment Jul 27 00:43:18.382694435 (61h7m28.105257465s) [upload=58607]: Waiting for error group managing segments... Jul 27 00:43:18.527690174 (61h7m28.250253214s) [noise=true , address=45.91.116.4:4964 , piece_id=IUGZAUAYCKCNNUR4HYYFYPOXBG6LQVHFN7PKW4GS5DYPQ7U36Z7A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000 Jul 27 00:43:18.688061510 (61h7m28.41062455s) [noise=true , address=79.127.205.235:4682 , piece_id=FMIZ33DBQT7WRPVPM64I4RJ4DHBK76KMNJBMSRGOKWG7HR2TLQUA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:18.711893046 (61h7m28.434456086s) [noise=true , address=79.127.205.234:4605 , piece_id=ALN4KNCH372YA7LW42FGNYR2DGN2OIVV3PWEONVL75OL2RAG26JA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.274880602 (61h7m28.997443642s) [noise=true , address=77.90.31.19:4871 , piece_id=EFFU35UFHEL7O2EUCVE57IMEYZUPHTNJPSHFCSZFB25R3AAPQCDA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.275742762 (61h7m28.998305792s) [noise=true , address=79.127.219.38:4017 , piece_id=ULPEENFK3JRUXSFVRTWHG4HQOVEEMSBXSF6XGYOZOED3ESPB6PEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.286576821 (61h7m29.009139861s) [noise=true , address=104.36.82.13:4214 , piece_id=PCAMOZGRCEUOABMCJYX6HIILND72K2FEAS3CXMWEXM7XWLM7DCFQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:19.366148283 (61h7m29.088711313s) [noise=true , address=77.90.31.16:4743 , piece_id=6VV54J6Y5RWKJOO2XQVECYA3TH7ZJSCIPTZQRDSZ4E25QID2C6BQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:22.036480782 (61h7m31.759043822s) [noise=true , address=64.49.13.138:4482 , piece_id=XZ4WG5GJCLUQDYFVVLRUBNY3HD3JWJKZ2V62ZYHO3KOS6UML56EA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:25.971040099 (61h7m35.693603139s) [noise=true , address=77.90.31.11:4587 , piece_id=5OPM4F55HG3YQEKYUDP36AV4FRC3LKTYX4NES4EGMM5F7XGNHK3Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:27.815494799 (61h7m37.538057839s) [noise=true , address=79.127.201.215:4348 , piece_id=5HXGM2RI2QTUL3QYI6APW6B76T6W7NPP5W6NXJFAGGTW7S3QSO4Q, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:43:32.498382741 (61h7m42.220945781s) [noise=true , address=79.127.201.215:4343 , piece_id=DIGWZ7KAMSAMI5X3BB7LXQKZIIKU64IBENXC47FWNCCO4IZMH54A, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:10.038797890 (61h8m19.76136093s) [noise=true , address=104.36.82.53:4128 , piece_id=4VJW3DARVMOTSNWFUHMZ6YOWU5223357RSGSIFOAA6S7QRCB2ERQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.735604254 (61h8m24.458167294s) [noise=true , address=104.36.82.56:4261 , piece_id=EPLPG6P3FEK6BULEVCABEU4OHJVR34OT4BLD7XXWPNLBZRZV47LQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.849222117 (61h8m24.571785147s) [noise=true , address=104.36.82.46:4489 , piece_id=YXSRA7M5VEJJNLMXLBFSKJRDKURHPSIBIWGPNIOJDBQEUPXCATMQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986831813 (61h8m24.709394853s) [noise=true , address=104.36.82.52:4112 , piece_id=EUK77E5E32L6Z5D6LLXUJNVHYOKKDM2AYYBBA65CVSL6D7XBKBTA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: <nil> Jul 27 00:44:14.986861119 (61h8m24.709424159s) [seg_pos={0 0}, upload=58607]: Segment reached optimal threshold of 14 pieces. Jul 27 00:44:14.987195659 (61h8m24.709758699s) [noise=true , address=104.36.82.15:4276 , piece_id=5GF3PFYQVLLOSXQNRUTPA6DSLE3TO3ZNPVZ2UHCW3IQUP373MECQ, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23): protocol: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32960->104.36.82.15:4276: use of closed network connection Jul 27 00:44:14.987245745 (61h8m24.709808785s) [noise=true , address=104.36.82.42:4811 , piece_id=SPS44UK6JWIHCAOXF4EVMF7BZBOFJWE3NS5DEQ3KVYC5DZBNIQEA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3): protocol: rpc: dial tcp 104.36.82.42:4811: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987293476 (61h8m24.709856506s) [noise=true , address=104.36.82.10:4096 , piece_id=VQOMJWACI4YAIXKZH3U3CLYIVNP3QKD3VJYGUN2GMIQ6MZ2KZRIA, seg_pos={0 0}, upload=58607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA): protocol: rpc: dial tcp 104.36.82.10:4096: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:44:14.987392385 (61h8m24.709955425s) [seg_pos={0 0}, upload=58607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:44:14.987531581 (61h8m24.710094621s) [upload=58607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:14.987537082 (61h8m24.710100112s) [upload=58607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:15.265313308 (61h8m24.987876348s) [upload=58607]: Done uploading segments... Jul 27 00:44:15.265332195 (61h8m24.987895235s) [upload=58607]: Upload finished. err: <nil> Duration: 1m0.002651076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [X.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX..] duration: 5.990545205s FAIL (2025-07-27 00:44:16 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.990572557s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.X.XX...X...............XX......X...XXX..XXX..X..X.X........] duration: 5.677749798s FAIL (2025-07-27 00:44:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.677772321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.] duration: 6.991306945s FAIL (2025-07-27 00:44:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== loading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:53950->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:07.609111953 (61h7m17.331674993s) [seg_pos={0 0}, upload=58605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:43:07.609198008 (61h7m17.331761038s) [upload=58605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:43:07.609202506 (61h7m17.331765546s) [upload=58605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:43:07.896913209 (61h7m17.619476249s) [upload=58605]: Done uploading segments... Jul 27 00:43:07.896931484 (61h7m17.619494524s) [upload=58605]: Upload finished. err: <nil> Jul 27 00:43:28.060523287 (61h7m37.783086327s) [upload=58612]: Starting upload Jul 27 00:43:28.060583983 (61h7m37.783147013s) [upload=58612]: Done starting upload Jul 27 00:43:28.060670480 (61h7m37.78323352s) [upload=58612]: Uploading segments... Jul 27 00:43:28.060757105 (61h7m37.783320135s) [upload=58612]: Got next segment. Inline: false Jul 27 00:43:28.060782454 (61h7m37.783345484s) [upload=58612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:28.060787423 (61h7m37.783350463s) [upload=58612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:28.290711414 (61h7m38.013274534s) [seg_pos={0 0}, upload=58612]: Begin upload segment... Jul 27 00:43:28.290781437 (61h7m38.013344487s) [seg_pos={0 0}, upload=58612]: Done begin upload segment. Jul 27 00:43:28.290806114 (61h7m38.013369154s) [noise=true , address=45.91.116.4:4940 , piece_id=MA45FUZBITCY6UR2CJFIEQGBZW2ZITG4AIED54LXO2SMMHJMPHJA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.290862703 (61h7m38.013425763s) [noise=true , address=45.91.116.6:4975 , piece_id=M2FD6AJSLFQLE4LCUG7EL5MP7LVBRA4CB2M4N3TADFQSFJV7QWPQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291198386 (61h7m38.013761426s) [noise=true , address=104.36.82.2:4732 , piece_id=HFCA7UMY5YMFNDI4F26QTR6DXTD3XZBOQV4BZBBRCK7SQMBDS2YQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291275082 (61h7m38.013838132s) [noise=true , address=77.90.31.13:4628 , piece_id=BLOWAKSVHXH7KUJVXZY6ZRX5MZWH3SUEQR2OO4Z6JKNCBGNM7VUQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291495654 (61h7m38.014058704s) [noise=true , address=45.91.116.14:4191 , piece_id=CHDM6NXETFACDXIUPRKQPYWJD4WDQDRDQOMHGF3KASZ5KDF2UHJQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291856834 (61h7m38.014419864s) [noise=true , address=45.91.116.13:4155 , piece_id=L2EBEYRWI4UZTNIHKRNCDY4ZI6Y5ZRKZCG4VUKRBSFCE6PQED7UA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291880410 (61h7m38.01444347s) [noise=true , address=104.36.82.12:4149 , piece_id=JVI4QFQMSNIJNX2LEWWDHZ3RPSMGY63326OXOFIQCZOCXM2DTN7A, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292322225 (61h7m38.014885265s) [noise=true , address=104.36.82.9:4005 , piece_id=XNVTMCWLML6MZLHYO7AF2N35B6FJZBUOH5T6WPG42Y4I5XCUHSZA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292451232 (61h7m38.015014282s) [noise=true , address=104.36.82.7:4911 , piece_id=DTUXFUXCMBDBNPMPS6FVF4DWLKZWKAX2MNAB36EZTHHVJ3F6ORSQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292763219 (61h7m38.015326259s) [noise=true , address=45.91.116.13:4153 , piece_id=MWCNTPBJRJFE4NVHAXNIH7BV3K3UD3O7Q6NIXFQ3YQYKUWEGVB6Q, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292937924 (61h7m38.015500964s) [noise=true , address=45.91.116.8:4045 , piece_id=ZYU4527GKVC6ZIDR4AH2TXGAN6VC63QFKHOJZJNQQ32FBAGTXOAQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293180598 (61h7m38.015743638s) [noise=true , address=104.36.82.5:4840 , piece_id=YTP7X34D2JFIN23IMU2FG2PRDJZMVG4RNDW2LUATWPQW3YF2REVQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293528122 (61h7m38.016091172s) [noise=true , address=104.36.82.13:4186 , piece_id=5BZ75HYCFMRX5XSUHPCG3KBWHQ7TXF5ZLBZSTKF2OLHI7O7CE52Q, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293599289 (61h7m38.016162329s) [noise=true , address=77.90.31.16:4733 , piece_id=UN56NU4BYL3ZJCYRTL3V4FXNE2E6XIOXQL6I3X2LTUBVF3GJIBYQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.294088004 (61h7m38.016651044s) [noise=true , address=104.36.82.10:4077 , piece_id=IC6VVDBP4VMLHV57GCLNSTXXAY44QT2WMV7Y733IC56CPAZ4ZDAA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.298318578 (61h7m38.020881608s) [noise=true , address=77.90.31.11:4555 , piece_id=VACZCFH53LME27B77R5P6JJLPWTXGFJ5E3UDYEEWFKIKHRKJDYVA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:30.041308206 (61h7m39.763871246s) [upload=58612]: Next returned nil segment Jul 27 00:43:30.041324637 (61h7m39.763887677s) [upload=58612]: Waiting for error group managing segments... Jul 27 00:43:30.349307817 (61h7m40.071870857s) [noise=true , address=104.36.82.9:4005 , piece_id=XNVTMCWLML6MZLHYO7AF2N35B6FJZBUOH5T6WPG42Y4I5XCUHSZA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.375421520 (61h7m40.09798456s) [noise=true , address=45.91.116.4:4940 , piece_id=MA45FUZBITCY6UR2CJFIEQGBZW2ZITG4AIED54LXO2SMMHJMPHJA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.381080437 (61h7m40.103643477s) [noise=true , address=45.91.116.13:4153 , piece_id=MWCNTPBJRJFE4NVHAXNIH7BV3K3UD3O7Q6NIXFQ3YQYKUWEGVB6Q, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.411307720 (61h7m40.13387076s) [noise=true , address=45.91.116.8:4045 , piece_id=ZYU4527GKVC6ZIDR4AH2TXGAN6VC63QFKHOJZJNQQ32FBAGTXOAQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.454531159 (61h7m40.177094199s) [noise=true , address=45.91.116.14:4191 , piece_id=CHDM6NXETFACDXIUPRKQPYWJD4WDQDRDQOMHGF3KASZ5KDF2UHJQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.499012215 (61h7m40.221575255s) [noise=true , address=45.91.116.13:4155 , piece_id=L2EBEYRWI4UZTNIHKRNCDY4ZI6Y5ZRKZCG4VUKRBSFCE6PQED7UA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.522553385 (61h7m40.245116425s) [noise=true , address=104.36.82.2:4732 , piece_id=HFCA7UMY5YMFNDI4F26QTR6DXTD3XZBOQV4BZBBRCK7SQMBDS2YQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.553051998 (61h7m40.275615038s) [noise=true , address=45.91.116.6:4975 , piece_id=M2FD6AJSLFQLE4LCUG7EL5MP7LVBRA4CB2M4N3TADFQSFJV7QWPQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:31.247387600 (61h7m40.96995063s) [noise=true , address=104.36.82.5:4840 , piece_id=YTP7X34D2JFIN23IMU2FG2PRDJZMVG4RNDW2LUATWPQW3YF2REVQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:32.097194573 (61h7m41.819757653s) [noise=true , address=77.90.31.16:4733 , piece_id=UN56NU4BYL3ZJCYRTL3V4FXNE2E6XIOXQL6I3X2LTUBVF3GJIBYQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:32.214252718 (61h7m41.936815758s) [noise=true , address=77.90.31.11:4555 , piece_id=VACZCFH53LME27B77R5P6JJLPWTXGFJ5E3UDYEEWFKIKHRKJDYVA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:57.025053300 (61h8m6.74761634s) [noise=true , address=77.90.31.13:4628 , piece_id=BLOWAKSVHXH7KUJVXZY6ZRX5MZWH3SUEQR2OO4Z6JKNCBGNM7VUQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:57.418787467 (61h8m7.141350507s) [noise=true , address=104.36.82.10:4077 , piece_id=IC6VVDBP4VMLHV57GCLNSTXXAY44QT2WMV7Y733IC56CPAZ4ZDAA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:35220->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418925340 (61h8m7.14148838s) [noise=true , address=104.36.82.7:4911 , piece_id=DTUXFUXCMBDBNPMPS6FVF4DWLKZWKAX2MNAB36EZTHHVJ3F6ORSQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:55892->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418975867 (61h8m7.141538907s) [noise=true , address=104.36.82.13:4186 , piece_id=5BZ75HYCFMRX5XSUHPCG3KBWHQ7TXF5ZLBZSTKF2OLHI7O7CE52Q, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:52952->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418980546 (61h8m7.141543586s) [noise=true , address=104.36.82.12:4149 , piece_id=JVI4QFQMSNIJNX2LEWWDHZ3RPSMGY63326OXOFIQCZOCXM2DTN7A, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:43978->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.419029480 (61h8m7.14159256s) [seg_pos={0 0}, upload=58612]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:43:57.648786891 (61h8m7.371349931s) [upload=58612]: Done uploading segments... Jul 27 00:43:57.648814473 (61h8m7.371377514s) [upload=58612]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.991327305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.977039922s FAIL (2025-07-27 00:44:02 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.977067905s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X] duration: 30.232289548s FAIL (2025-07-27 00:43:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== loading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:53950->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:07.609111953 (61h7m17.331674993s) [seg_pos={0 0}, upload=58605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:43:07.609198008 (61h7m17.331761038s) [upload=58605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:43:07.609202506 (61h7m17.331765546s) [upload=58605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:43:07.896913209 (61h7m17.619476249s) [upload=58605]: Done uploading segments... Jul 27 00:43:07.896931484 (61h7m17.619494524s) [upload=58605]: Upload finished. err: <nil> Jul 27 00:43:28.060523287 (61h7m37.783086327s) [upload=58612]: Starting upload Jul 27 00:43:28.060583983 (61h7m37.783147013s) [upload=58612]: Done starting upload Jul 27 00:43:28.060670480 (61h7m37.78323352s) [upload=58612]: Uploading segments... Jul 27 00:43:28.060757105 (61h7m37.783320135s) [upload=58612]: Got next segment. Inline: false Jul 27 00:43:28.060782454 (61h7m37.783345484s) [upload=58612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:43:28.060787423 (61h7m37.783350463s) [upload=58612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:43:28.290711414 (61h7m38.013274534s) [seg_pos={0 0}, upload=58612]: Begin upload segment... Jul 27 00:43:28.290781437 (61h7m38.013344487s) [seg_pos={0 0}, upload=58612]: Done begin upload segment. Jul 27 00:43:28.290806114 (61h7m38.013369154s) [noise=true , address=45.91.116.4:4940 , piece_id=MA45FUZBITCY6UR2CJFIEQGBZW2ZITG4AIED54LXO2SMMHJMPHJA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.290862703 (61h7m38.013425763s) [noise=true , address=45.91.116.6:4975 , piece_id=M2FD6AJSLFQLE4LCUG7EL5MP7LVBRA4CB2M4N3TADFQSFJV7QWPQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291198386 (61h7m38.013761426s) [noise=true , address=104.36.82.2:4732 , piece_id=HFCA7UMY5YMFNDI4F26QTR6DXTD3XZBOQV4BZBBRCK7SQMBDS2YQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291275082 (61h7m38.013838132s) [noise=true , address=77.90.31.13:4628 , piece_id=BLOWAKSVHXH7KUJVXZY6ZRX5MZWH3SUEQR2OO4Z6JKNCBGNM7VUQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291495654 (61h7m38.014058704s) [noise=true , address=45.91.116.14:4191 , piece_id=CHDM6NXETFACDXIUPRKQPYWJD4WDQDRDQOMHGF3KASZ5KDF2UHJQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291856834 (61h7m38.014419864s) [noise=true , address=45.91.116.13:4155 , piece_id=L2EBEYRWI4UZTNIHKRNCDY4ZI6Y5ZRKZCG4VUKRBSFCE6PQED7UA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.291880410 (61h7m38.01444347s) [noise=true , address=104.36.82.12:4149 , piece_id=JVI4QFQMSNIJNX2LEWWDHZ3RPSMGY63326OXOFIQCZOCXM2DTN7A, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292322225 (61h7m38.014885265s) [noise=true , address=104.36.82.9:4005 , piece_id=XNVTMCWLML6MZLHYO7AF2N35B6FJZBUOH5T6WPG42Y4I5XCUHSZA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292451232 (61h7m38.015014282s) [noise=true , address=104.36.82.7:4911 , piece_id=DTUXFUXCMBDBNPMPS6FVF4DWLKZWKAX2MNAB36EZTHHVJ3F6ORSQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292763219 (61h7m38.015326259s) [noise=true , address=45.91.116.13:4153 , piece_id=MWCNTPBJRJFE4NVHAXNIH7BV3K3UD3O7Q6NIXFQ3YQYKUWEGVB6Q, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.292937924 (61h7m38.015500964s) [noise=true , address=45.91.116.8:4045 , piece_id=ZYU4527GKVC6ZIDR4AH2TXGAN6VC63QFKHOJZJNQQ32FBAGTXOAQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293180598 (61h7m38.015743638s) [noise=true , address=104.36.82.5:4840 , piece_id=YTP7X34D2JFIN23IMU2FG2PRDJZMVG4RNDW2LUATWPQW3YF2REVQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293528122 (61h7m38.016091172s) [noise=true , address=104.36.82.13:4186 , piece_id=5BZ75HYCFMRX5XSUHPCG3KBWHQ7TXF5ZLBZSTKF2OLHI7O7CE52Q, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.293599289 (61h7m38.016162329s) [noise=true , address=77.90.31.16:4733 , piece_id=UN56NU4BYL3ZJCYRTL3V4FXNE2E6XIOXQL6I3X2LTUBVF3GJIBYQ, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.294088004 (61h7m38.016651044s) [noise=true , address=104.36.82.10:4077 , piece_id=IC6VVDBP4VMLHV57GCLNSTXXAY44QT2WMV7Y733IC56CPAZ4ZDAA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:28.298318578 (61h7m38.020881608s) [noise=true , address=77.90.31.11:4555 , piece_id=VACZCFH53LME27B77R5P6JJLPWTXGFJ5E3UDYEEWFKIKHRKJDYVA, seg_pos={0 0}, upload=58612]: Uploading piece... Jul 27 00:43:30.041308206 (61h7m39.763871246s) [upload=58612]: Next returned nil segment Jul 27 00:43:30.041324637 (61h7m39.763887677s) [upload=58612]: Waiting for error group managing segments... Jul 27 00:43:30.349307817 (61h7m40.071870857s) [noise=true , address=104.36.82.9:4005 , piece_id=XNVTMCWLML6MZLHYO7AF2N35B6FJZBUOH5T6WPG42Y4I5XCUHSZA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.375421520 (61h7m40.09798456s) [noise=true , address=45.91.116.4:4940 , piece_id=MA45FUZBITCY6UR2CJFIEQGBZW2ZITG4AIED54LXO2SMMHJMPHJA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.381080437 (61h7m40.103643477s) [noise=true , address=45.91.116.13:4153 , piece_id=MWCNTPBJRJFE4NVHAXNIH7BV3K3UD3O7Q6NIXFQ3YQYKUWEGVB6Q, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.411307720 (61h7m40.13387076s) [noise=true , address=45.91.116.8:4045 , piece_id=ZYU4527GKVC6ZIDR4AH2TXGAN6VC63QFKHOJZJNQQ32FBAGTXOAQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.454531159 (61h7m40.177094199s) [noise=true , address=45.91.116.14:4191 , piece_id=CHDM6NXETFACDXIUPRKQPYWJD4WDQDRDQOMHGF3KASZ5KDF2UHJQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.499012215 (61h7m40.221575255s) [noise=true , address=45.91.116.13:4155 , piece_id=L2EBEYRWI4UZTNIHKRNCDY4ZI6Y5ZRKZCG4VUKRBSFCE6PQED7UA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.522553385 (61h7m40.245116425s) [noise=true , address=104.36.82.2:4732 , piece_id=HFCA7UMY5YMFNDI4F26QTR6DXTD3XZBOQV4BZBBRCK7SQMBDS2YQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:30.553051998 (61h7m40.275615038s) [noise=true , address=45.91.116.6:4975 , piece_id=M2FD6AJSLFQLE4LCUG7EL5MP7LVBRA4CB2M4N3TADFQSFJV7QWPQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:31.247387600 (61h7m40.96995063s) [noise=true , address=104.36.82.5:4840 , piece_id=YTP7X34D2JFIN23IMU2FG2PRDJZMVG4RNDW2LUATWPQW3YF2REVQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:32.097194573 (61h7m41.819757653s) [noise=true , address=77.90.31.16:4733 , piece_id=UN56NU4BYL3ZJCYRTL3V4FXNE2E6XIOXQL6I3X2LTUBVF3GJIBYQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:32.214252718 (61h7m41.936815758s) [noise=true , address=77.90.31.11:4555 , piece_id=VACZCFH53LME27B77R5P6JJLPWTXGFJ5E3UDYEEWFKIKHRKJDYVA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:57.025053300 (61h8m6.74761634s) [noise=true , address=77.90.31.13:4628 , piece_id=BLOWAKSVHXH7KUJVXZY6ZRX5MZWH3SUEQR2OO4Z6JKNCBGNM7VUQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: <nil> Jul 27 00:43:57.418787467 (61h8m7.141350507s) [noise=true , address=104.36.82.10:4077 , piece_id=IC6VVDBP4VMLHV57GCLNSTXXAY44QT2WMV7Y733IC56CPAZ4ZDAA, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:35220->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418925340 (61h8m7.14148838s) [noise=true , address=104.36.82.7:4911 , piece_id=DTUXFUXCMBDBNPMPS6FVF4DWLKZWKAX2MNAB36EZTHHVJ3F6ORSQ, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:55892->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418975867 (61h8m7.141538907s) [noise=true , address=104.36.82.13:4186 , piece_id=5BZ75HYCFMRX5XSUHPCG3KBWHQ7TXF5ZLBZSTKF2OLHI7O7CE52Q, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:52952->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.418980546 (61h8m7.141543586s) [noise=true , address=104.36.82.12:4149 , piece_id=JVI4QFQMSNIJNX2LEWWDHZ3RPSMGY63326OXOFIQCZOCXM2DTN7A, seg_pos={0 0}, upload=58612]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:43978->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:57.419029480 (61h8m7.14159256s) [seg_pos={0 0}, upload=58612]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:43:57.648786891 (61h8m7.371349931s) [upload=58612]: Done uploading segments... Jul 27 00:43:57.648814473 (61h8m7.371377514s) [upload=58612]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232306892s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...X......X.XX...XXXX.....X..........X..X..X...XX..........] duration: 7.529788539s FAIL (2025-07-27 00:43:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.529813285s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.] duration: 7.020350848s FAIL (2025-07-27 00:43:49 UTC) orbiter warning: uplink: context deadline exceeded Duration: 7.020436762s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X...X......X.XX...XXXX.....X..........X..X..X...XX.........] duration: 5.83313763s FAIL (2025-07-27 00:43:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.833160594s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [.XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX] duration: 30.002426766s FAIL (2025-07-27 00:43:15 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.002452816s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.] duration: 5.839658075s FAIL (2025-07-27 00:43:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== true , address=104.36.82.2:4732 , piece_id=X4V7K4DDWYN5ZOYJE4PG6MRZJKRYIDUHCHZDNVOEDK62AXCUE2SQ, seg_pos={0 0}, upload=58575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 188.166.225.170:49186->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:08.618647277 (61h5m18.341210317s) [seg_pos={0 0}, upload=58575]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:41:08.618706270 (61h5m18.34126931s) [upload=58575]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:41:08.618709977 (61h5m18.341273017s) [upload=58575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:41:09.066971850 (61h5m18.78953489s) [upload=58575]: Done uploading segments... Jul 27 00:41:09.066994283 (61h5m18.789557323s) [upload=58575]: Upload finished. err: <nil> Jul 27 00:41:59.867159718 (61h6m9.589722768s) [upload=58588]: Starting upload Jul 27 00:41:59.867245562 (61h6m9.589808602s) [upload=58588]: Done starting upload Jul 27 00:41:59.867321347 (61h6m9.589884387s) [upload=58588]: Uploading segments... Jul 27 00:41:59.867524265 (61h6m9.590087295s) [upload=58588]: Got next segment. Inline: false Jul 27 00:41:59.867586214 (61h6m9.590149264s) [upload=58588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:41:59.867591905 (61h6m9.590154945s) [upload=58588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:42:00.102737386 (61h6m9.825300416s) [seg_pos={0 0}, upload=58588]: Begin upload segment... Jul 27 00:42:00.102779776 (61h6m9.825342816s) [seg_pos={0 0}, upload=58588]: Done begin upload segment. Jul 27 00:42:00.102801278 (61h6m9.825364308s) [noise=true , address=45.91.116.11:4118 , piece_id=Y7X4JWZJDR7673NAAJEW7AW4D5HBE5UDQZFAEYT3OSUPOQEZ6RRQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.102846094 (61h6m9.825409124s) [noise=true , address=45.91.116.8:4047 , piece_id=JTASFHBGU4ALZ3F2BO3J2W46OVZGCTTUPYTJWEBB7VJB5YPFUZ4Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103240107 (61h6m9.825803147s) [noise=true , address=104.36.82.9:4006 , piece_id=IXAUGIN7SN2UA5G7O56AYUMV7KSPDVBCOSWOSQTHMNCYAQE3OHOQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103336531 (61h6m9.825899571s) [noise=true , address=45.91.116.5:4297 , piece_id=U5ZBQMSIGQLWSFM4ERMSFRVNCLZBRFXYTBK5DKHOJK75RRVQUNYA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103580899 (61h6m9.826143939s) [noise=true , address=45.91.116.12:4227 , piece_id=LAPCOILKWIS3YRC4R5DRB5GJGNKK7TMWQZXWEKQZGIINYP63PABA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103650994 (61h6m9.826214034s) [noise=true , address=104.36.82.10:4077 , piece_id=U4TG6LC2HQCI3HWLC3XKVE7H4IT7XMP4Y4IUNKOKNGQXMPE4ICVQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103936319 (61h6m9.826499359s) [noise=true , address=104.36.82.7:4912 , piece_id=PBATRG2H3HPS25WQ4EWIL3VK5KEHR7POWQ2ANLTDGQGM655AF2IQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104330203 (61h6m9.826893243s) [noise=true , address=45.91.116.8:4048 , piece_id=FKLORIVXS26GKT6AWI6PPCHQHG2ZHVOR4ECP6VV7XSHN5CRNVSMA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104636319 (61h6m9.827199349s) [noise=true , address=45.91.116.6:4975 , piece_id=U72QW54AKT36V3TX2KJLPWLJNLZHOZ7V3JUVA4C476P6XGNWQK6Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104885456 (61h6m9.827448496s) [noise=true , address=45.91.116.6:4974 , piece_id=HPHY7H7SLZSPI2WYW6S7UWH2TG2FSISYH4B6DZLGUW5E4IFBY36A, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105121977 (61h6m9.827685017s) [noise=true , address=77.90.31.16:4734 , piece_id=LBMV5TY5UWZHPOFGMTYBCXWB2SZMXJJLQWL7FBQS64ZIOBRKRNAA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105330807 (61h6m9.827893847s) [noise=true , address=45.91.116.4:4939 , piece_id=NO4NNGTSP3OJMUUT2MQILGWTL7SS4PKVXQ7BEMNUDLPBVSDVLBCA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105423625 (61h6m9.827986665s) [noise=true , address=45.91.116.7:4010 , piece_id=G6VUQWOEP5SYSI6MNB3ZRNZLGBEDBBGIUFFAWOZFQPUKNJGNPLEQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105691517 (61h6m9.828254557s) [noise=true , address=77.90.31.14:4661 , piece_id=NIYBWLRWGOQTBWBC43CX34OU3OP7UQRZE7A7347TT4IFUKQODG3Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105926467 (61h6m9.828489507s) [noise=true , address=104.36.82.6:4875 , piece_id=UWUY6K3AWUMSKJYT3ETNLAKYKO2YU4FFQJH7FTSOCR5F2EAFOCJA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105996781 (61h6m9.828559821s) [noise=true , address=45.91.116.14:4189 , piece_id=T5HPBLJDCN6XYRB3YE4KCWJY7JIOTWSX5H3NDY4JDRKY3Q4BWIKA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:01.832554780 (61h6m11.55511782s) [upload=58588]: Next returned nil segment Jul 27 00:42:01.832578957 (61h6m11.555142007s) [upload=58588]: Waiting for error group managing segments... Jul 27 00:42:02.113654143 (61h6m11.836217183s) [noise=true , address=45.91.116.6:4974 , piece_id=HPHY7H7SLZSPI2WYW6S7UWH2TG2FSISYH4B6DZLGUW5E4IFBY36A, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.113875226 (61h6m11.836438266s) [noise=true , address=45.91.116.7:4010 , piece_id=G6VUQWOEP5SYSI6MNB3ZRNZLGBEDBBGIUFFAWOZFQPUKNJGNPLEQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.122043925 (61h6m11.844606966s) [noise=true , address=45.91.116.5:4297 , piece_id=U5ZBQMSIGQLWSFM4ERMSFRVNCLZBRFXYTBK5DKHOJK75RRVQUNYA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.144804754 (61h6m11.867367804s) [noise=true , address=45.91.116.12:4227 , piece_id=LAPCOILKWIS3YRC4R5DRB5GJGNKK7TMWQZXWEKQZGIINYP63PABA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.194302797 (61h6m11.916865837s) [noise=true , address=77.90.31.16:4734 , piece_id=LBMV5TY5UWZHPOFGMTYBCXWB2SZMXJJLQWL7FBQS64ZIOBRKRNAA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.212140827 (61h6m11.934703867s) [noise=true , address=45.91.116.6:4975 , piece_id=U72QW54AKT36V3TX2KJLPWLJNLZHOZ7V3JUVA4C476P6XGNWQK6Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.215132661 (61h6m11.937695701s) [noise=true , address=45.91.116.8:4048 , piece_id=FKLORIVXS26GKT6AWI6PPCHQHG2ZHVOR4ECP6VV7XSHN5CRNVSMA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.226526110 (61h6m11.94908915s) [noise=true , address=45.91.116.4:4939 , piece_id=NO4NNGTSP3OJMUUT2MQILGWTL7SS4PKVXQ7BEMNUDLPBVSDVLBCA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.239060171 (61h6m11.961623211s) [noise=true , address=45.91.116.14:4189 , piece_id=T5HPBLJDCN6XYRB3YE4KCWJY7JIOTWSX5H3NDY4JDRKY3Q4BWIKA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.287304216 (61h6m12.009867256s) [noise=true , address=77.90.31.14:4661 , piece_id=NIYBWLRWGOQTBWBC43CX34OU3OP7UQRZE7A7347TT4IFUKQODG3Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.290839160 (61h6m12.01340221s) [noise=true , address=45.91.116.8:4047 , piece_id=JTASFHBGU4ALZ3F2BO3J2W46OVZGCTTUPYTJWEBB7VJB5YPFUZ4Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.298282962 (61h6m12.020846002s) [noise=true , address=45.91.116.11:4118 , piece_id=Y7X4JWZJDR7673NAAJEW7AW4D5HBE5UDQZFAEYT3OSUPOQEZ6RRQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:29.221210346 (61h6m38.943773386s) [noise=true , address=104.36.82.9:4006 , piece_id=IXAUGIN7SN2UA5G7O56AYUMV7KSPDVBCOSWOSQTHMNCYAQE3OHOQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221257307 (61h6m38.943820337s) [noise=true , address=104.36.82.7:4912 , piece_id=PBATRG2H3HPS25WQ4EWIL3VK5KEHR7POWQ2ANLTDGQGM655AF2IQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221291121 (61h6m38.943854161s) [noise=true , address=104.36.82.10:4077 , piece_id=U4TG6LC2HQCI3HWLC3XKVE7H4IT7XMP4Y4IUNKOKNGQXMPE4ICVQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221325297 (61h6m38.943888337s) [noise=true , address=104.36.82.6:4875 , piece_id=UWUY6K3AWUMSKJYT3ETNLAKYKO2YU4FFQJH7FTSOCR5F2EAFOCJA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:50082->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221339103 (61h6m38.943902133s) [seg_pos={0 0}, upload=58588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:42:29.461689803 (61h6m39.184252843s) [upload=58588]: Done uploading segments... Jul 27 00:42:29.461705864 (61h6m39.184268904s) [upload=58588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.839683474s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233775466s FAIL (2025-07-27 00:43:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Done begin upload segment. Jul 27 00:42:04.785142655 (61h6m14.507705695s) [noise=true , address=104.36.82.10:4100 , piece_id=O46ZDV4F643AVLSDFAJIXVUTXC3JSCJU4FVZR34HYVOTPK42T26Q, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785202089 (61h6m14.507765129s) [noise=true , address=104.36.82.56:4258 , piece_id=76V3RW55QWH3Y3CH3IC5OCXBE6PI6W2CSTQ6C7GLYIT72ZP2HMUA, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785252966 (61h6m14.507816006s) [noise=true , address=79.127.205.246:4262 , piece_id=LSFXKXWXK4Z5VORQS7IFOLSSNGXLJHPZZPP72ZH4VJML2MEI6VZQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785552720 (61h6m14.50811575s) [noise=true , address=104.36.82.46:4511 , piece_id=R3AVBVU3PPOOTT2IGTH3IDRZHCHTSZXHSZHRL4I5X3AYWXMWGSXQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785648724 (61h6m14.508211754s) [noise=true , address=79.127.219.38:4001 , piece_id=XAPBQVMSKM5WY734J7QKJHIMLLGEHZEDU63FGC3PDIVZJVFC53EQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785840571 (61h6m14.508403611s) [noise=true , address=104.36.82.54:4176 , piece_id=VUMPR5JXGY3QJ4UEAIA3WXFWOYJIGIBBAIHT2WQM6EVJJZZZF7XQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.785932957 (61h6m14.508495997s) [noise=true , address=79.127.219.38:4015 , piece_id=DCGZ2CELQA2ZJLI32IRCYECTTCGAOG5XQYZSQAFLMDQQ76ORSPWQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.786180421 (61h6m14.508743461s) [noise=true , address=64.49.13.139:4478 , piece_id=74U7AHILBJTTWXCSEFNQJ34SPTMPDXGDBM364ZTJ24XRQ7YNI3IA, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.786436851 (61h6m14.508999891s) [noise=true , address=77.90.31.18:4808 , piece_id=FHMJB43JS663HBVDMKBIIDOQW35WM76BU6HG7AAZMA4WAG3FQ5FQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.786588912 (61h6m14.509151963s) [noise=true , address=79.127.219.38:4009 , piece_id=4CMD3MFUPNXO7DS4JSCWS774WQ5YB753IAJZEVQEWAMSWQ2PWF3Q, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.786676690 (61h6m14.50923973s) [noise=true , address=104.36.82.52:4093 , piece_id=5EASC4U5X2LOW2LMLQAGZBNKXA7PW352AXE6DPF7NNSVC5ZXHAGA, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.786915047 (61h6m14.509478087s) [noise=true , address=104.36.82.53:4134 , piece_id=Q2ORRC2BJEM5LBGTTRIU4HXL73I7XVDVYUGDOJULWJUSDCTJK2AQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787046859 (61h6m14.509609889s) [noise=true , address=104.36.82.55:4224 , piece_id=JLNXUOBATLASOFJY2ZRDHBM4NSXTC4QRTUYJQZNWKP57NYXXRCMA, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787106002 (61h6m14.509669042s) [noise=true , address=104.36.82.47:4521 , piece_id=JXILHMKJ4O6KRHLSEJQUYQ5Y2GCITGOO6NV7NJTKJJEQVFY3644A, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787310122 (61h6m14.509873162s) [noise=true , address=104.36.82.4:4819 , piece_id=ID5J72EHY47UDSEALG7GAON3Z24IBU53QEOZIFN6CNKKP4HR3BRA, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787518351 (61h6m14.510081391s) [noise=true , address=79.127.205.245:4240 , piece_id=X2YYCYYNWHIFEYQRAQ5OH3QRSJY62I4F35IQD5VMLZV4GZQAKRBQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787716440 (61h6m14.51027948s) [noise=true , address=156.146.43.225:4448 , piece_id=T5PUJDQF6XXDGTYDYFT3GK5BBGJEH6737BRBOHDHNKDLSLZP7KBQ, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:04.787962471 (61h6m14.510525511s) [noise=true , address=79.127.205.235:4686 , piece_id=6T7TIYC56I6XKHFQAK6FXGS3NVBHSPK5VIPDXQ22YB234VGDN33Q, seg_pos={0 0}, upload=58589]: Uploading piece... Jul 27 00:42:05.821517206 (61h6m15.544080236s) [noise=true , address=156.146.43.225:4448 , piece_id=T5PUJDQF6XXDGTYDYFT3GK5BBGJEH6737BRBOHDHNKDLSLZP7KBQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:42:05.899180068 (61h6m15.621743108s) [upload=58589]: Next returned nil segment Jul 27 00:42:05.899197260 (61h6m15.6217603s) [upload=58589]: Waiting for error group managing segments... Jul 27 00:42:06.186741615 (61h6m15.909304655s) [noise=true , address=79.127.205.235:4686 , piece_id=6T7TIYC56I6XKHFQAK6FXGS3NVBHSPK5VIPDXQ22YB234VGDN33Q, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:06.200602645 (61h6m15.923165685s) [noise=true , address=79.127.205.246:4262 , piece_id=LSFXKXWXK4Z5VORQS7IFOLSSNGXLJHPZZPP72ZH4VJML2MEI6VZQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:06.230780924 (61h6m15.953343984s) [noise=true , address=79.127.205.245:4240 , piece_id=X2YYCYYNWHIFEYQRAQ5OH3QRSJY62I4F35IQD5VMLZV4GZQAKRBQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:06.697146484 (61h6m16.419709524s) [noise=true , address=77.90.31.18:4808 , piece_id=FHMJB43JS663HBVDMKBIIDOQW35WM76BU6HG7AAZMA4WAG3FQ5FQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:06.753883168 (61h6m16.476446208s) [noise=true , address=79.127.219.38:4015 , piece_id=DCGZ2CELQA2ZJLI32IRCYECTTCGAOG5XQYZSQAFLMDQQ76ORSPWQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:06.961489417 (61h6m16.684052457s) [noise=true , address=79.127.219.38:4001 , piece_id=XAPBQVMSKM5WY734J7QKJHIMLLGEHZEDU63FGC3PDIVZJVFC53EQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:07.002187174 (61h6m16.724750224s) [noise=true , address=79.127.219.38:4009 , piece_id=4CMD3MFUPNXO7DS4JSCWS774WQ5YB753IAJZEVQEWAMSWQ2PWF3Q, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:42:16.006986552 (61h6m25.729549622s) [noise=true , address=64.49.13.139:4478 , piece_id=74U7AHILBJTTWXCSEFNQJ34SPTMPDXGDBM364ZTJ24XRQ7YNI3IA, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:43:00.424439591 (61h7m10.147002622s) [noise=true , address=104.36.82.46:4511 , piece_id=R3AVBVU3PPOOTT2IGTH3IDRZHCHTSZXHSZHRL4I5X3AYWXMWGSXQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:43:00.688562063 (61h7m10.411125103s) [noise=true , address=104.36.82.47:4521 , piece_id=JXILHMKJ4O6KRHLSEJQUYQ5Y2GCITGOO6NV7NJTKJJEQVFY3644A, seg_pos={0 0}, upload=58589]: Done uploading piece. err: <nil> Jul 27 00:43:03.618547383 (61h7m13.341110423s) [noise=true , address=104.36.82.52:4093 , piece_id=5EASC4U5X2LOW2LMLQAGZBNKXA7PW352AXE6DPF7NNSVC5ZXHAGA, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: rpc: dial tcp 104.36.82.52:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618612206 (61h7m13.341175236s) [noise=true , address=104.36.82.4:4819 , piece_id=ID5J72EHY47UDSEALG7GAON3Z24IBU53QEOZIFN6CNKKP4HR3BRA, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:12mh9u38YmD7NcPA5uB1KLaw8dToTUKywJsi1MhhSc17LXo15Gb, address:104.36.82.4:4819): protocol: write tcp 188.166.225.170:39594->104.36.82.4:4819: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618641412 (61h7m13.341204452s) [noise=true , address=104.36.82.56:4258 , piece_id=76V3RW55QWH3Y3CH3IC5OCXBE6PI6W2CSTQ6C7GLYIT72ZP2HMUA, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:12g7DdMjQHyYTD9hRvixfNedxyPr6q6uE2rLdEbhQ1MGANRMDm3, address:104.36.82.56:4258): protocol: write tcp 188.166.225.170:45148->104.36.82.56:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618671069 (61h7m13.341234109s) [noise=true , address=104.36.82.54:4176 , piece_id=VUMPR5JXGY3QJ4UEAIA3WXFWOYJIGIBBAIHT2WQM6EVJJZZZF7XQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: rpc: dial tcp 104.36.82.54:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618710875 (61h7m13.341273915s) [noise=true , address=104.36.82.53:4134 , piece_id=Q2ORRC2BJEM5LBGTTRIU4HXL73I7XVDVYUGDOJULWJUSDCTJK2AQ, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: rpc: dial tcp 104.36.82.53:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618720934 (61h7m13.341283974s) [noise=true , address=104.36.82.55:4224 , piece_id=JLNXUOBATLASOFJY2ZRDHBM4NSXTC4QRTUYJQZNWKP57NYXXRCMA, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: rpc: dial tcp 104.36.82.55:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618730884 (61h7m13.341293924s) [noise=true , address=104.36.82.10:4100 , piece_id=O46ZDV4F643AVLSDFAJIXVUTXC3JSCJU4FVZR34HYVOTPK42T26Q, seg_pos={0 0}, upload=58589]: Done uploading piece. err: ecclient: upload failed (node:12Hyc6sbtuPXMLyFzubMcepbJyLyVAGNpkXAmqg73i3sY97dwdW, address:104.36.82.10:4100): protocol: rpc: dial tcp 104.36.82.10:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:43:03.618744980 (61h7m13.34130802s) [seg_pos={0 0}, upload=58589]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:43:03.850086514 (61h7m13.572649554s) [upload=58589]: Done uploading segments... Jul 27 00:43:03.850111922 (61h7m13.572674962s) [upload=58589]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.233791669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.852844591s FAIL (2025-07-27 00:43:01 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.852877474s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X.X.XX...X...............XX......X...XXX..XXX..X..X.X.......] duration: 5.486130424s FAIL (2025-07-27 00:43:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.486180611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [XX.X.XX...X...............XX......X...XXX..XXX..X..X.X......] duration: 16.542392525s FAIL (2025-07-27 00:42:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.542416792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXX] duration: 30.26796102s FAIL (2025-07-27 00:42:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.267984837s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X...X......X.XX...XXXX.....X..........X..X..X...XX........] duration: 5.938656788s FAIL (2025-07-27 00:42:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.938679451s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X] duration: 30.242376621s FAIL (2025-07-27 00:42:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== true , address=104.36.82.2:4732 , piece_id=X4V7K4DDWYN5ZOYJE4PG6MRZJKRYIDUHCHZDNVOEDK62AXCUE2SQ, seg_pos={0 0}, upload=58575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 188.166.225.170:49186->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:08.618647277 (61h5m18.341210317s) [seg_pos={0 0}, upload=58575]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:41:08.618706270 (61h5m18.34126931s) [upload=58575]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:41:08.618709977 (61h5m18.341273017s) [upload=58575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:41:09.066971850 (61h5m18.78953489s) [upload=58575]: Done uploading segments... Jul 27 00:41:09.066994283 (61h5m18.789557323s) [upload=58575]: Upload finished. err: <nil> Jul 27 00:41:59.867159718 (61h6m9.589722768s) [upload=58588]: Starting upload Jul 27 00:41:59.867245562 (61h6m9.589808602s) [upload=58588]: Done starting upload Jul 27 00:41:59.867321347 (61h6m9.589884387s) [upload=58588]: Uploading segments... Jul 27 00:41:59.867524265 (61h6m9.590087295s) [upload=58588]: Got next segment. Inline: false Jul 27 00:41:59.867586214 (61h6m9.590149264s) [upload=58588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:41:59.867591905 (61h6m9.590154945s) [upload=58588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:42:00.102737386 (61h6m9.825300416s) [seg_pos={0 0}, upload=58588]: Begin upload segment... Jul 27 00:42:00.102779776 (61h6m9.825342816s) [seg_pos={0 0}, upload=58588]: Done begin upload segment. Jul 27 00:42:00.102801278 (61h6m9.825364308s) [noise=true , address=45.91.116.11:4118 , piece_id=Y7X4JWZJDR7673NAAJEW7AW4D5HBE5UDQZFAEYT3OSUPOQEZ6RRQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.102846094 (61h6m9.825409124s) [noise=true , address=45.91.116.8:4047 , piece_id=JTASFHBGU4ALZ3F2BO3J2W46OVZGCTTUPYTJWEBB7VJB5YPFUZ4Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103240107 (61h6m9.825803147s) [noise=true , address=104.36.82.9:4006 , piece_id=IXAUGIN7SN2UA5G7O56AYUMV7KSPDVBCOSWOSQTHMNCYAQE3OHOQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103336531 (61h6m9.825899571s) [noise=true , address=45.91.116.5:4297 , piece_id=U5ZBQMSIGQLWSFM4ERMSFRVNCLZBRFXYTBK5DKHOJK75RRVQUNYA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103580899 (61h6m9.826143939s) [noise=true , address=45.91.116.12:4227 , piece_id=LAPCOILKWIS3YRC4R5DRB5GJGNKK7TMWQZXWEKQZGIINYP63PABA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103650994 (61h6m9.826214034s) [noise=true , address=104.36.82.10:4077 , piece_id=U4TG6LC2HQCI3HWLC3XKVE7H4IT7XMP4Y4IUNKOKNGQXMPE4ICVQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.103936319 (61h6m9.826499359s) [noise=true , address=104.36.82.7:4912 , piece_id=PBATRG2H3HPS25WQ4EWIL3VK5KEHR7POWQ2ANLTDGQGM655AF2IQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104330203 (61h6m9.826893243s) [noise=true , address=45.91.116.8:4048 , piece_id=FKLORIVXS26GKT6AWI6PPCHQHG2ZHVOR4ECP6VV7XSHN5CRNVSMA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104636319 (61h6m9.827199349s) [noise=true , address=45.91.116.6:4975 , piece_id=U72QW54AKT36V3TX2KJLPWLJNLZHOZ7V3JUVA4C476P6XGNWQK6Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.104885456 (61h6m9.827448496s) [noise=true , address=45.91.116.6:4974 , piece_id=HPHY7H7SLZSPI2WYW6S7UWH2TG2FSISYH4B6DZLGUW5E4IFBY36A, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105121977 (61h6m9.827685017s) [noise=true , address=77.90.31.16:4734 , piece_id=LBMV5TY5UWZHPOFGMTYBCXWB2SZMXJJLQWL7FBQS64ZIOBRKRNAA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105330807 (61h6m9.827893847s) [noise=true , address=45.91.116.4:4939 , piece_id=NO4NNGTSP3OJMUUT2MQILGWTL7SS4PKVXQ7BEMNUDLPBVSDVLBCA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105423625 (61h6m9.827986665s) [noise=true , address=45.91.116.7:4010 , piece_id=G6VUQWOEP5SYSI6MNB3ZRNZLGBEDBBGIUFFAWOZFQPUKNJGNPLEQ, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105691517 (61h6m9.828254557s) [noise=true , address=77.90.31.14:4661 , piece_id=NIYBWLRWGOQTBWBC43CX34OU3OP7UQRZE7A7347TT4IFUKQODG3Q, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105926467 (61h6m9.828489507s) [noise=true , address=104.36.82.6:4875 , piece_id=UWUY6K3AWUMSKJYT3ETNLAKYKO2YU4FFQJH7FTSOCR5F2EAFOCJA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:00.105996781 (61h6m9.828559821s) [noise=true , address=45.91.116.14:4189 , piece_id=T5HPBLJDCN6XYRB3YE4KCWJY7JIOTWSX5H3NDY4JDRKY3Q4BWIKA, seg_pos={0 0}, upload=58588]: Uploading piece... Jul 27 00:42:01.832554780 (61h6m11.55511782s) [upload=58588]: Next returned nil segment Jul 27 00:42:01.832578957 (61h6m11.555142007s) [upload=58588]: Waiting for error group managing segments... Jul 27 00:42:02.113654143 (61h6m11.836217183s) [noise=true , address=45.91.116.6:4974 , piece_id=HPHY7H7SLZSPI2WYW6S7UWH2TG2FSISYH4B6DZLGUW5E4IFBY36A, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.113875226 (61h6m11.836438266s) [noise=true , address=45.91.116.7:4010 , piece_id=G6VUQWOEP5SYSI6MNB3ZRNZLGBEDBBGIUFFAWOZFQPUKNJGNPLEQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.122043925 (61h6m11.844606966s) [noise=true , address=45.91.116.5:4297 , piece_id=U5ZBQMSIGQLWSFM4ERMSFRVNCLZBRFXYTBK5DKHOJK75RRVQUNYA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.144804754 (61h6m11.867367804s) [noise=true , address=45.91.116.12:4227 , piece_id=LAPCOILKWIS3YRC4R5DRB5GJGNKK7TMWQZXWEKQZGIINYP63PABA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.194302797 (61h6m11.916865837s) [noise=true , address=77.90.31.16:4734 , piece_id=LBMV5TY5UWZHPOFGMTYBCXWB2SZMXJJLQWL7FBQS64ZIOBRKRNAA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.212140827 (61h6m11.934703867s) [noise=true , address=45.91.116.6:4975 , piece_id=U72QW54AKT36V3TX2KJLPWLJNLZHOZ7V3JUVA4C476P6XGNWQK6Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.215132661 (61h6m11.937695701s) [noise=true , address=45.91.116.8:4048 , piece_id=FKLORIVXS26GKT6AWI6PPCHQHG2ZHVOR4ECP6VV7XSHN5CRNVSMA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.226526110 (61h6m11.94908915s) [noise=true , address=45.91.116.4:4939 , piece_id=NO4NNGTSP3OJMUUT2MQILGWTL7SS4PKVXQ7BEMNUDLPBVSDVLBCA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.239060171 (61h6m11.961623211s) [noise=true , address=45.91.116.14:4189 , piece_id=T5HPBLJDCN6XYRB3YE4KCWJY7JIOTWSX5H3NDY4JDRKY3Q4BWIKA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.287304216 (61h6m12.009867256s) [noise=true , address=77.90.31.14:4661 , piece_id=NIYBWLRWGOQTBWBC43CX34OU3OP7UQRZE7A7347TT4IFUKQODG3Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.290839160 (61h6m12.01340221s) [noise=true , address=45.91.116.8:4047 , piece_id=JTASFHBGU4ALZ3F2BO3J2W46OVZGCTTUPYTJWEBB7VJB5YPFUZ4Q, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:02.298282962 (61h6m12.020846002s) [noise=true , address=45.91.116.11:4118 , piece_id=Y7X4JWZJDR7673NAAJEW7AW4D5HBE5UDQZFAEYT3OSUPOQEZ6RRQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: <nil> Jul 27 00:42:29.221210346 (61h6m38.943773386s) [noise=true , address=104.36.82.9:4006 , piece_id=IXAUGIN7SN2UA5G7O56AYUMV7KSPDVBCOSWOSQTHMNCYAQE3OHOQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221257307 (61h6m38.943820337s) [noise=true , address=104.36.82.7:4912 , piece_id=PBATRG2H3HPS25WQ4EWIL3VK5KEHR7POWQ2ANLTDGQGM655AF2IQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221291121 (61h6m38.943854161s) [noise=true , address=104.36.82.10:4077 , piece_id=U4TG6LC2HQCI3HWLC3XKVE7H4IT7XMP4Y4IUNKOKNGQXMPE4ICVQ, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221325297 (61h6m38.943888337s) [noise=true , address=104.36.82.6:4875 , piece_id=UWUY6K3AWUMSKJYT3ETNLAKYKO2YU4FFQJH7FTSOCR5F2EAFOCJA, seg_pos={0 0}, upload=58588]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:50082->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:29.221339103 (61h6m38.943902133s) [seg_pos={0 0}, upload=58588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:42:29.461689803 (61h6m39.184252843s) [upload=58588]: Done uploading segments... Jul 27 00:42:29.461705864 (61h6m39.184268904s) [upload=58588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.242393824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.859516064s FAIL (2025-07-27 00:42:18 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.859549629s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230154452s FAIL (2025-07-27 00:42:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 9.13.26:4953 , piece_id=5Y4IGZPA6IVL3KBNBI4AMVRZ4LBMVNAMB6ZESZ5QUGUHUEAHSS4A, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.281505415 (61h5m14.004068455s) [noise=true , address=77.90.31.17:4798 , piece_id=NBI6QAZEMYHWCD5Q7PIHK3VKDFMCKZEG5CCCKQ2ZQEZ6ZHZIJ3RA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.281590748 (61h5m14.004153788s) [noise=true , address=156.146.43.225:4449 , piece_id=CEP3OYAT54MORXPOFR7J7ZSZ2VVS4HX6EJXF7PAJ34ASULHAERUA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.281779188 (61h5m14.004342229s) [noise=true , address=79.127.205.233:4652 , piece_id=DQAKEWUAPGO7LIYGWB4GPNGRAXAQDAQEQFYIORV5VFPL6RNCDHTQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.282059434 (61h5m14.004622475s) [noise=true , address=104.36.82.47:4533 , piece_id=AS5YO7FDM5LCMAK2JBJF6F5YJUPMS6E2P3U5TE5ATTETYJH3KMWA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.282342457 (61h5m14.004905497s) [noise=true , address=79.127.201.215:4358 , piece_id=B5A2TJE7DNKIB3OOILCUGJ5R6MT4T34MYUDELYUVQQTIHGG4GNQQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.282631549 (61h5m14.005194589s) [noise=true , address=77.90.31.18:4808 , piece_id=RZLEG2RVUXSMQTVXPN4B6NGDD5ZB7DWWTDIIQ4NKJJ2BOUQZ5S2A, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.282854095 (61h5m14.005417135s) [noise=true , address=77.90.31.21:4906 , piece_id=EGX353TKU2ZDXQ4NBOINI3OP7FDJU4OFNHWV3NKFXSIKPJV5E4QA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.283059278 (61h5m14.005622318s) [noise=true , address=104.36.82.46:4511 , piece_id=VNUMKF4OMIJRIH4UP2URJPZ4I2I7YZ62RW6I4UYVO5Y54GRFCHVA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.283279299 (61h5m14.005842329s) [noise=true , address=104.36.82.53:4121 , piece_id=NJNJ4DJEZ54JW3SE74WOMPGSOTITSCF2CLWNB36MHWBSC63FMQGA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.283496164 (61h5m14.006059194s) [noise=true , address=79.127.201.215:4356 , piece_id=J26JGG4UOVETDXUVYRJR742W7FYEPZS2G2PMF2OO5WWVFN75LPNQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.283696136 (61h5m14.006259177s) [noise=true , address=77.90.31.16:4746 , piece_id=5XL5RQKYILX3SST5ZRW6BOVMDPHBWIWK6D25HKMXBH2XQREKH3GQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.283952477 (61h5m14.006515507s) [noise=true , address=79.127.205.235:4676 , piece_id=DL6OOBP52DR5H2MLV5JC6GVL2RTSMBBAPJJN2QCSWRWEF5ET3IPA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.284157159 (61h5m14.006720199s) [noise=true , address=79.127.205.245:4242 , piece_id=GW3T62S6N6UHKIAEICIUTYDZD7SEPTYIIFGF2MCZZORKGJK26IKA, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.284369995 (61h5m14.006933025s) [noise=true , address=104.36.82.54:4177 , piece_id=VUJ3AA7YIPUMKUFZBYVWGJQ5O3WKMQ5NZQGTHL4XSUSGZ7OVSVNQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.284578745 (61h5m14.007141775s) [noise=true , address=104.36.82.12:4173 , piece_id=UUTD5HXRZRQ4EQRGAL7BFBDCFZCPJVJLNKL5LUPEOZ54XYBNSO3A, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:04.284795059 (61h5m14.007358099s) [noise=true , address=104.36.82.50:4048 , piece_id=OCDELTG5CJVE5ZLFP73LIZCHOFOGFOFOVR3TQTXO5TI2DGXNU2KQ, seg_pos={0 0}, upload=58574]: Uploading piece... Jul 27 00:41:05.386679138 (61h5m15.109242178s) [upload=58574]: Next returned nil segment Jul 27 00:41:05.386699126 (61h5m15.109262156s) [upload=58574]: Waiting for error group managing segments... Jul 27 00:41:05.673420807 (61h5m15.395983847s) [noise=true , address=79.127.205.245:4242 , piece_id=GW3T62S6N6UHKIAEICIUTYDZD7SEPTYIIFGF2MCZZORKGJK26IKA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:05.751378121 (61h5m15.473941201s) [noise=true , address=79.127.205.235:4676 , piece_id=DL6OOBP52DR5H2MLV5JC6GVL2RTSMBBAPJJN2QCSWRWEF5ET3IPA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:05.785236732 (61h5m15.507799772s) [noise=true , address=79.127.205.233:4652 , piece_id=DQAKEWUAPGO7LIYGWB4GPNGRAXAQDAQEQFYIORV5VFPL6RNCDHTQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:05.963974118 (61h5m15.686537158s) [noise=true , address=64.49.13.26:4953 , piece_id=5Y4IGZPA6IVL3KBNBI4AMVRZ4LBMVNAMB6ZESZ5QUGUHUEAHSS4A, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:06.245235561 (61h5m15.967798601s) [noise=true , address=77.90.31.18:4808 , piece_id=RZLEG2RVUXSMQTVXPN4B6NGDD5ZB7DWWTDIIQ4NKJJ2BOUQZ5S2A, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:06.313042727 (61h5m16.035605767s) [noise=true , address=77.90.31.11:4562 , piece_id=CBZEYSFVV3US7ZO5UJQLBKRHUTX67IQEGOTMLFM2G2S5KOVF7RHA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:06.384514823 (61h5m16.107077863s) [noise=true , address=77.90.31.16:4746 , piece_id=5XL5RQKYILX3SST5ZRW6BOVMDPHBWIWK6D25HKMXBH2XQREKH3GQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:08.671465203 (61h5m18.394028243s) [noise=true , address=156.146.43.225:4449 , piece_id=CEP3OYAT54MORXPOFR7J7ZSZ2VVS4HX6EJXF7PAJ34ASULHAERUA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:16.014149055 (61h5m25.736712155s) [noise=true , address=79.127.201.215:4358 , piece_id=B5A2TJE7DNKIB3OOILCUGJ5R6MT4T34MYUDELYUVQQTIHGG4GNQQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:41:20.053999614 (61h5m29.776562654s) [noise=true , address=79.127.201.215:4356 , piece_id=J26JGG4UOVETDXUVYRJR742W7FYEPZS2G2PMF2OO5WWVFN75LPNQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: <nil> Jul 27 00:42:03.167926679 (61h6m12.890489719s) [noise=true , address=104.36.82.47:4533 , piece_id=AS5YO7FDM5LCMAK2JBJF6F5YJUPMS6E2P3U5TE5ATTETYJH3KMWA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: write tcp 188.166.225.170:45696->104.36.82.47:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168008936 (61h6m12.890572006s) [noise=true , address=104.36.82.46:4511 , piece_id=VNUMKF4OMIJRIH4UP2URJPZ4I2I7YZ62RW6I4UYVO5Y54GRFCHVA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6, address:104.36.82.46:4511): protocol: write tcp 188.166.225.170:50204->104.36.82.46:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168075384 (61h6m12.890638444s) [noise=true , address=77.90.31.21:4906 , piece_id=EGX353TKU2ZDXQ4NBOINI3OP7FDJU4OFNHWV3NKFXSIKPJV5E4QA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: write tcp 188.166.225.170:35840->77.90.31.21:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168094470 (61h6m12.89065752s) [noise=true , address=77.90.31.17:4798 , piece_id=NBI6QAZEMYHWCD5Q7PIHK3VKDFMCKZEG5CCCKQ2ZQEZ6ZHZIJ3RA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: write tcp 188.166.225.170:58378->77.90.31.17:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168115911 (61h6m12.890678971s) [noise=true , address=104.36.82.50:4048 , piece_id=OCDELTG5CJVE5ZLFP73LIZCHOFOGFOFOVR3TQTXO5TI2DGXNU2KQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:1rJd5FahWjKfvJstcoLE5uJGgtVj2uZirPpCJxbfGPGQ3j4qxC, address:104.36.82.50:4048): protocol: write tcp 188.166.225.170:35764->104.36.82.50:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168159725 (61h6m12.890722775s) [noise=true , address=104.36.82.53:4121 , piece_id=NJNJ4DJEZ54JW3SE74WOMPGSOTITSCF2CLWNB36MHWBSC63FMQGA, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: write tcp 188.166.225.170:58594->104.36.82.53:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168180704 (61h6m12.890743774s) [noise=true , address=104.36.82.12:4173 , piece_id=UUTD5HXRZRQ4EQRGAL7BFBDCFZCPJVJLNKL5LUPEOZ54XYBNSO3A, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:12HbGUhwH2eNqY75sLQCDgkvJ1QSrJ2n4Qg8A5Z73dFXxRpVwSC, address:104.36.82.12:4173): protocol: write tcp 188.166.225.170:59106->104.36.82.12:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168226292 (61h6m12.890789332s) [noise=true , address=104.36.82.54:4177 , piece_id=VUJ3AA7YIPUMKUFZBYVWGJQ5O3WKMQ5NZQGTHL4XSUSGZ7OVSVNQ, seg_pos={0 0}, upload=58574]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 188.166.225.170:56836->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:03.168255929 (61h6m12.890818969s) [seg_pos={0 0}, upload=58574]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:42:03.395637014 (61h6m13.118200054s) [upload=58574]: Done uploading segments... Jul 27 00:42:03.395661221 (61h6m13.118224261s) [upload=58574]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230167266s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)