Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 11.941062314s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:30:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 11.941093114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.127182817s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:29:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.127211272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.087827756s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:29:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.087858154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.927395958s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:28:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.927425285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.024208534s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:28:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.024231398s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.406949447s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:27:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.407054166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.272958765s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:26:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.272988192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X........................................X..................] duration: 6.133272014s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:25:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.133312861s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X........................................X.................] duration: 6.035610845s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:24:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.035639159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X........................................X................] duration: 6.088996531s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:23:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.089025746s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X........................................X...............] duration: 6.020606572s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:23:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.020628202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X........................................X..............] duration: 6.064232969s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:22:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.06426055s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X........................................X.............] duration: 6.090300287s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:21:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.090323853s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X........................................X............] duration: 6.10650178s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:21:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.106603704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X........................................X...........] duration: 6.129165761s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:20:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.129277483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X........................................X..........] duration: 6.239525219s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:19:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.239562169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X........................................X.........] duration: 6.090094295s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:18:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.090123954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X........................................X........] duration: 6.229374334s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:17:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.229422956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X........................................X.......] duration: 6.13016616s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:15:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.130196719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X........................................X......] duration: 5.990332095s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:14:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.990392691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X........................................X.....] duration: 6.006184001s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:13:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.006212166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X........................................X....] duration: 5.956109692s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:13:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.956168104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X........................................X...] duration: 5.960496442s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:11:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.960515728s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X........................................X..] duration: 6.080144358s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:11:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.08017067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X........................................X.] duration: 5.979252232s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:11:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.97928255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X........................................X] duration: 30.255406709s trace: https://jaeger.dev.storj.io/trace/7af2ac50f736eb31 FAIL (2025-11-15 03:10:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 15 03:09:53.466530503 (139h5m52.018000965s) [upload=122447]: Starting upload Nov 15 03:09:53.466588594 (139h5m52.018059055s) [upload=122447]: Done starting upload Nov 15 03:09:53.466797923 (139h5m52.018268384s) [upload=122447]: Uploading segments... Nov 15 03:09:53.466836597 (139h5m52.018307048s) [upload=122447]: Got next segment. Inline: false Nov 15 03:09:53.466861835 (139h5m52.018332296s) [upload=122447]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 03:09:53.466866203 (139h5m52.018336664s) [upload=122447]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 03:09:53.722817759 (139h5m52.27428821s) [seg_pos={0 0}, upload=122447]: Begin upload segment... Nov 15 03:09:53.722927027 (139h5m52.274397488s) [seg_pos={0 0}, upload=122447]: Done begin upload segment. Nov 15 03:09:53.723000367 (139h5m52.274470818s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723072935 (139h5m52.274543396s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.723821431 (139h5m52.275291882s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724123847 (139h5m52.275594298s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724413809 (139h5m52.27588427s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.724704482 (139h5m52.276174933s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725287152 (139h5m52.276757603s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.725912532 (139h5m52.277382983s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.726376706 (139h5m52.277847157s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.727604825 (139h5m52.279075286s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.728626701 (139h5m52.280097162s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.729247593 (139h5m52.280718054s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730018511 (139h5m52.281488962s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.730596261 (139h5m52.282066713s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731047681 (139h5m52.282518142s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:53.731903961 (139h5m52.283374422s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Uploading piece... Nov 15 03:09:55.460812822 (139h5m54.012283283s) [upload=122447]: Next returned nil segment Nov 15 03:09:55.460826177 (139h5m54.012296638s) [upload=122447]: Waiting for error group managing segments... Nov 15 03:09:55.790808844 (139h5m54.342279306s) [noise=true , address=45.91.116.8:4045 , piece_id=6IRCRYR5YWGXLLGS2L7VRG4AJOVSVVZ6Y7V3YZ3IEPI3EG73BPSQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.800617803 (139h5m54.352088264s) [noise=true , address=77.90.31.17:4772 , piece_id=S5CB2UASM225DDEAZITMEJQXAXDDY6AQK6VWH7XKCY3AWN3D6FEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.845721079 (139h5m54.39719154s) [noise=true , address=77.90.31.16:4734 , piece_id=L4RTD5K432DUPUD3LTCJOJ2DRVF5P3FF7QD5KZJDEHREQAXDY3EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.868970530 (139h5m54.420440992s) [noise=true , address=77.90.31.14:4662 , piece_id=3TQBUJU64DPOGFZE25OJ453HZKZXJXG2W2QBBBZPLMYUMPAYYATA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.869967279 (139h5m54.42143774s) [noise=true , address=45.91.116.5:4297 , piece_id=BQVI6FX3ZIAD6LTPADGUTDRTAEP6WXBSFYIM3B2XJ63FCTCIC2RA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.880797663 (139h5m54.432268114s) [noise=true , address=77.90.31.10:4518 , piece_id=TYVR63DHJQM42D57NODYFYXCWUW7GCUH2IHHJCXYCNEDRLIZUEJQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.883568318 (139h5m54.435038769s) [noise=true , address=45.91.116.12:4225 , piece_id=4HRY2J7YHV5K2GHVBQ2STSD6ALS33BPQIK2RVLTABV7QD6N7FV2A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.888538162 (139h5m54.440008623s) [noise=true , address=45.91.116.14:4189 , piece_id=GULBR2JVYTDUUTFVCJQEBU2AUAX77A5MDBMNLO5F65WCC33AR5EA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919811032 (139h5m54.471281493s) [noise=true , address=77.90.31.15:4697 , piece_id=QEIXGD3S36XJQMESKDPXJEM46S2U2VKOUISLFGB2RRZK7ZAL6EEA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:55.919956529 (139h5m54.47142699s) [noise=true , address=77.90.31.19:4841 , piece_id=P2GRPEIPG2ZCGBBBFB7NWVPJ5IP2SRI423OVYUGIWXKWJT6EK4JA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:09:56.024261328 (139h5m54.575731789s) [noise=true , address=77.90.31.21:4879 , piece_id=BV4B4B2QQ6N6DL42SU6452J37ZWGGW7JSR6MWU26QHUN5I4B4G6Q, seg_pos={0 0}, upload=122447]: Done uploading piece. err: <nil> Nov 15 03:10:22.751646914 (139h6m21.303117405s) [noise=true , address=145.79.227.2:4114 , piece_id=5XHL37ZNZ5MJJJJW7434RZQKSEMBMGEXNWUXHNC4KXCAHGAXGB4A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55866->145.79.227.2:4114: use of closed network connection Nov 15 03:10:22.751685317 (139h6m21.303155778s) [noise=true , address=145.79.227.30:4761 , piece_id=FBCFWRO3CR54Q4VEYD3X5V2TC32NMQU4ST5KX45ZWE644OALWMVQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:42530->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751896108 (139h6m21.303366569s) [noise=true , address=145.79.227.11:4730 , piece_id=NHY5INM7H2Y6SLQ5XWQ3OKVEQPIDDWJXWBPDHHFFCNHCHH3PKRRQ, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: read tcp 188.166.225.170:59390->145.79.227.11:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.751990668 (139h6m21.30346113s) [noise=true , address=145.79.227.7:4295 , piece_id=A6LV6MX7YFCKTIQUR4N3OTHIFQGRZYWLS6ABQ2BCHUUTMMNJ5Y3A, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: read tcp 188.166.225.170:47074->145.79.227.7:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752235394 (139h6m21.303705856s) [noise=true , address=145.79.227.25:4617 , piece_id=LPAAQES7VLYUZMJIF35UXI32U4T3WYCKFFVUJHOBX6VWR3RF5XWA, seg_pos={0 0}, upload=122447]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:43240->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 03:10:22.752274609 (139h6m21.30374507s) [seg_pos={0 0}, upload=122447]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 15 03:10:23.004520909 (139h6m21.55599137s) [upload=122447]: Done uploading segments... Nov 15 03:10:23.004560945 (139h6m21.556031406s) [upload=122447]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.255424934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.065784234s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:09:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.065821216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.999798241s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:08:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.999827897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.005462306s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:07:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.00549126s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.487326412s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:07:21 UTC) orbiter warning: context deadline exceeded Duration: 3.487364925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.166363136s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:06:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.16638633s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.459312839s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:06:26 UTC) orbiter warning: context deadline exceeded Duration: 3.459338428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.014683006s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:06:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.014712523s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.402989705s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:05:53 UTC) orbiter warning: context deadline exceeded Duration: 3.403017908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.123714004s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:05:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.123741656s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.526332141s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:05:08 UTC) orbiter warning: context deadline exceeded Duration: 3.52635309s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.531148634s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:04:44 UTC) orbiter warning: context deadline exceeded Duration: 3.53118925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.013783509s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:04:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.013803417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.137863614s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:03:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.137891857s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.445100171s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:03:30 UTC) orbiter warning: context deadline exceeded Duration: 3.445132032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.120165853s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:02:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.120202262s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.464032663s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:02:24 UTC) orbiter warning: context deadline exceeded Duration: 3.464060305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.166089511s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:02:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.16612093s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.436944391s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:01:48 UTC) orbiter warning: context deadline exceeded Duration: 3.436973866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.056156168s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:01:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.056184792s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.544403955s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:01:02 UTC) orbiter warning: context deadline exceeded Duration: 3.544450915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.018760628s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 03:00:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.01880944s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.481668753s trace: https://jaeger.dev.storj.io/trace/71a8a0ae48722667 FAIL (2025-11-15 03:00:27 UTC) orbiter warning: context deadline exceeded Duration: 3.481697657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.012519732s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:59:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.012543307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.012145559s trace: https://jaeger.dev.storj.io/trace/279540056f3c8f5a FAIL (2025-11-15 02:59:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== , upload=122004]: Uploading piece... Nov 15 02:39:24.054620843 (138h35m22.606091294s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.055790660 (138h35m22.607261121s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056150475 (138h35m22.607620936s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056488488 (138h35m22.607958939s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.056793830 (138h35m22.608264291s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.057181217 (138h35m22.608651669s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058468318 (138h35m22.609938769s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.058968020 (138h35m22.610438471s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.060301610 (138h35m22.611772081s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.061672419 (138h35m22.613142881s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.063539074 (138h35m22.615009535s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.064965641 (138h35m22.616436092s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:24.066458323 (138h35m22.617928784s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Uploading piece... Nov 15 02:39:25.850109277 (138h35m24.401579738s) [upload=122004]: Next returned nil segment Nov 15 02:39:25.850129356 (138h35m24.401599807s) [upload=122004]: Waiting for error group managing segments... Nov 15 02:39:26.142180193 (138h35m24.693650654s) [noise=true , address=45.91.116.6:4973 , piece_id=WUHE4SYQC2JJ6LQVEVA6HU7BJI4W2YKSXXDO6RJGGMGQZWAEYVMQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.168440246 (138h35m24.719910698s) [noise=true , address=77.90.31.19:4842 , piece_id=OZW6URETKXPPTLB4F7YB5YX73VXVUIKUISAYUSIA2NFXSIGEUZKQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.187342260 (138h35m24.738812711s) [noise=true , address=45.91.116.10:4082 , piece_id=PP4CQZGHCLZT22XN4GCGDYRWXQRYEYX4C3CPY3BCEF5QKNQ4OA7Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.192534166 (138h35m24.744004627s) [noise=true , address=77.90.31.13:4627 , piece_id=OW6Z6EI5VEOYURIYABW5DLZBLUSM72HCTE6LLC5D4URB6RMTE7KA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.194847622 (138h35m24.746318083s) [noise=true , address=77.90.31.17:4771 , piece_id=T7NXU2ATZBW2YCJAU6KSXAPD4JGFSEZL6TIXJ6K24VKZGFJDBAXA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.216500162 (138h35m24.767970613s) [noise=true , address=77.90.31.16:4734 , piece_id=MNCLURTDYIQRUEFA5MP7PV436ZLMH7SR4RTUP4KRKU3DNYNXQQBQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:26.263219524 (138h35m24.814689985s) [noise=true , address=45.91.116.5:4297 , piece_id=UDQ2S6IGLFESCFGSVB2TDKPM6AJNVICV66UTPQ2NSVDKXTYQ6QHQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: <nil> Nov 15 02:39:53.089780545 (138h35m51.641251007s) [noise=true , address=145.79.227.24:4582 , piece_id=ORGTANCTI5VFZAA72UNJ5C2AHF65HWMQQST4OCTRWJZ2BGSYI54Q, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53812->145.79.227.24:4582: use of closed network connection Nov 15 02:39:53.090073292 (138h35m51.641543754s) [noise=true , address=145.79.227.9:4367 , piece_id=M463SXNI4YQIY6BXMVBBHHTPQ3KB2U5E5XOKSSL3SQ5G3KXLXNVQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45842->145.79.227.9:4367: use of closed network connection Nov 15 02:39:53.090099272 (138h35m51.641569743s) [noise=true , address=145.79.227.15:4875 , piece_id=2EO4OVSBS6FWBGPAY3NAEEU4F6UKL6DMFGLQPIFJ6RV4Z76YYLQQ, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58816->145.79.227.15:4875: use of closed network connection Nov 15 02:39:53.090220373 (138h35m51.641690844s) [noise=true , address=145.79.227.20:4437 , piece_id=EBKK54TTS5KV7YX7T3HA2INFSQCQIZ7Z7OFNGONKQEUEE5OMSXUA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39120->145.79.227.20:4437: use of closed network connection Nov 15 02:39:53.090343557 (138h35m51.641814018s) [noise=true , address=145.79.227.28:4727 , piece_id=5CZNATHSQERVIMZLOF5QZNWSIHFSFPH6EY3NKEEBWTVKDG34ZUWA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090515805 (138h35m51.641986276s) [noise=true , address=145.79.227.14:4838 , piece_id=HWFWU3BPPPDVSF6JS4G4ZQ7E3OSZH3AV4CEJNM4PJESVTQOI4MFA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45008->145.79.227.14:4838: use of closed network connection Nov 15 02:39:53.090692201 (138h35m51.642162682s) [noise=true , address=145.79.227.18:4005 , piece_id=OSR2Q4L7XD7CG6ODY4Z3FNRUZPEPMIG7VMNHWWPTY7KRCHMYKQ3A, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:40256->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 02:39:53.090864089 (138h35m51.64233455s) [noise=true , address=145.79.227.16:4909 , piece_id=SBZP77VAM3UB4EO7QNSRCPKDYVMPYEQSDNAVARSHSLB2SSXW3KZA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39308->145.79.227.16:4909: use of closed network connection Nov 15 02:39:53.091151496 (138h35m51.642621967s) [noise=true , address=145.79.227.11:4732 , piece_id=JR74D2NCEMGIPFNMBSZRJXXS64DLG34DDDL4TCVZHLPA4PLQ5HSA, seg_pos={0 0}, upload=122004]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:58736->145.79.227.11:4732: use of closed network connection; context deadline exceeded Nov 15 02:39:53.091188225 (138h35m51.642658677s) [seg_pos={0 0}, upload=122004]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 02:39:53.342967760 (138h35m51.894438221s) [upload=122004]: Done uploading segments... Nov 15 02:39:53.343078511 (138h35m51.894548972s) [upload=122004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.012180044s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)